this post was submitted on 08 Jul 2023
218 points (98.7% liked)

Selfhosted

40750 readers
795 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud
Loki
CannaVet
devve
HybridSarcasm
[email protected]
218
Introducing XPipe: A brand-new type of shell connection hub and remote file manager (sh.itjust.works)
submitted 1 year ago by [email protected] to c/selfhosted
49 comments fedilink hide all child comments
 

Hello there Lemmy users, I recently posted an announcement of my project on the selfhosted subreddit and I think it is a good idea to also post it here for the Lemmy users.

About this project

I always wanted to have an easy file system and terminal access to all of my servers, including containers and clusters that you normally can't connect to with existing solutions out of the box. So over the last months I worked on my new project XPipe to fix that.

In short, it is a brand-new type of shell connection hub with an included remote file manager that works by only interacting with already installed command-line tools on local and remote shell connections. This approach makes it much more flexible as it doesn't have to deal with file system APIs, protocols, or libraries at all, everything is delegated to your own CLI tools. This also allows you to open connections in your favorite terminal application through XPipe. So if you normally use CLI tools like ssh, docker, kubectl, etc. to connect to your servers, you can just use XPipe on top of that without any setup required on your servers.

Here are some screenshots:

Shell Connection Hub

File Manager

In the context of the selfhosted community, the application is technically not hosted as it is implemented as a desktop application to have access to your shells,command-line programs, and terminals, but you can use it to access all your self-hosted infrastructure. The application matches the spirit of selfhosted as you have full control over your data. Everything is stored on your system, it doesn't need to connect to any online service and there are no accounts or anything like that. It is also designed to be cross-platform and should also run on every operating system.

So if this project sounds interesting to you, you can give it a try! There are more features to come in the near future. I also appreciate any kind of bug reports and feedback to guide me in the right development direction. There is also a Discord and a Slack workspace for any sort of talking, although there isn't really a community yet. Any sort of issue reports are important as I only had the ability to test it in a few different server environments and your setups can differ wildly from mine.

Enjoy!

top 43 comments
sorted by: hot top controversial new old
[–] raffomania 8 points 1 year ago (1 children)

Looks interesting! What are some of the things you use this for?

[–] [email protected] 3 points 1 year ago (2 children)

You can use it to work with any remote shell connections. I use it for my WSL instances on Windows and Docker containers on my Linux servers to connect to them in my terminal and interact with the file system through the graphical user interface as I prefer managing files that way. Support has also been expanded by request to include things like Kubernetes clusters and their containers.

Overall, XPipe makes it much less tedious to connect and access remote systems wherever they are located, especially if you have to go through multiple intermediate systems in between. Once you added a system to XPipe, you can just connect to it with your favorite terminal in one click and also browse the file system.

[–] indy 3 points 1 year ago

Overall, XPipe makes it much less tedious to connect and access remote systems wherever they are located, especially if you have to go through multiple intermediate systems in between.

Love the idea, and the support for multiple different platforms and application types gives it the potential to be a standard go-to tool. Thank you for making and sharing this!

[–] [email protected] 1 points 1 year ago (1 children)

Is there any plans to add support for podman containers or no?

[–] [email protected] 1 points 1 year ago

If I can somehow launch a shell process into a container in podman, that should be easy to add

[–] [email protected] 8 points 1 year ago* (last edited 1 year ago) (1 children)

I don't get it. Maybe I'm dumb... but:

If I am used to work with command line tools, then what do I need this GUI for?

[–] [email protected] 10 points 1 year ago

You get a fancy overview over all your remote connections and don't have to type anything to establish shell connections in your terminal, you get launched into the session with one click, so it gives you an overview over your server infrastructure and saves you some typing effort.

Also you can access the file system of any connected remote system via a graphical user interface, but I guess that is personal preference whether you would like to use something like this or not.

[–] [email protected] 5 points 1 year ago

This looks really awesome. While I manage my servers mostly through ssh, having the occasional file system mount just a click away would definitely be more convenient.

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago) (2 children)

I think my environments might be a bit too large for the app to handle. I have ~90 docker containers running on one of my servers and it seems to be really struggling with it it. Generally I've been having some performance issues (clicking on anything has a 1-3 second delay) which appear to be amplified by the number of active containers and clients.

Memory usage increases to infinity, this is a snapshot after launching the program and having it open a shell.
https://i.imgur.com/L0y2JFN.png

It's a really cool idea though and I like the UI and the ability to browse file systems via gui without having to map a network drive.

[–] [email protected] 3 points 1 year ago

An update: I was able to reproduce the issue of growing memory usage when frequently adding connections like containers. As long as you don't add more connections continuously in a session, the memory shouldn't really grow that much. So a restart should improve the situation.

[–] [email protected] 2 points 1 year ago (1 children)

That is unfortunate to hear about these performance issues. Performance was not my main focus initially as optimizations are always supposed to come later on, but I guess that time is now. How many connections do you have added in total? I was not able to reproduce anything getting over 1GB of main memory with like 50 total connections. Also, does restarting fix some of that?

The best way of diagnosing that issue would be a heap dump of the application, but that requires some effort of getting it and also sharing it somehow, but we could do that if you want.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

86 connections with 83 of those being active docker containers.

Edit: If you need anything for diagnostics I'll happily provide it.

Restarting does help but sometimes I still get 3GB spikes, but it drops back down after a while instead of increasing.

[–] [email protected] 3 points 1 year ago

Looks very promising. I'll give it a try. Thank you for your effort!

[–] flubba86 3 points 1 year ago* (last edited 1 year ago) (1 children)

Wow, I was looking for something exactly like this only last week.

I use alacritty as my terminal emulator, and I normally have three or four alacrity windows open to different remote hosts. I was thinking how good it would be to have a GUI tool that lists all my remotes and allows me to open alacritty directly to a SSH session on a given host, and I even thought it would be great if that GUI tool had file management capabilities, so I didn't need to have FileZilla open for each host too.

You've made my day. I'll try it out on Monday.

[–] [email protected] 3 points 1 year ago

That is great to hear! I just happened to very recently add built-in support for more Linux terminals like alacritty, so I hope everything will work here.

[–] [email protected] 3 points 1 year ago

Looks awesome! I'll def be checking this out. Thanks!

[–] Spazztastik 3 points 1 year ago

Wow installed it a few mins ago and I can already see this as a huge time saver for a few things I do day to day! with a few added features I can see myself fully switching to this from mRemoteNG!

[–] abeltramo 3 points 1 year ago (1 children)

Sounds super interesting, thanks for sharing! I'll definitely check it out later!

Quick question: are both docker and fs based on top of ssh or are there any more requirements? For example, do you expect the docker socket to be available over the network or do you open an ssh connection and then access the docker socket from there?

[–] [email protected] 1 points 1 year ago

Nothing is really based on top of ssh, it is just one supported way of connecting to remote systems. The docker socket does not have to be available in the network here. You can first open an SSH connection to the host on which the docker containers and socket are located on, and from there connect to the containers.

[–] [email protected] 2 points 1 year ago (3 children)

Any chance of adding this to the AUR?

[–] [email protected] 2 points 1 year ago

That would definitely be possible, although I have no idea on how to do that so I would have to familiarize myself with the process first.

[–] [email protected] 2 points 1 year ago

I managed to do that, so you can now try it out at: https://aur.archlinux.org/packages/xpipe. Let me know whether everything works for you there, I was only able to test it on one Manjaro VM.

[–] [email protected] 1 points 1 year ago (1 children)

I would also appreciate this, the project looks great!

[–] [email protected] 2 points 1 year ago

I managed to do that, so you can now try it out at: https://aur.archlinux.org/packages/xpipe. Let me know whether everything works for you there, I was only able to test it on one Manjaro VM.

[–] [email protected] 2 points 1 year ago

Thank you for thinking to include a portable version. My first thought was that I'd need to request that and wait for it way down the line

[–] [email protected] 2 points 1 year ago (1 children)

This is really interesting. I wonder, does it support, or will it at some point support, any form of ssh tunneling?

I have a scenario where we have a number of Linux KVM hypervisors that are accessible via zerotier connections (they're behind firewalls that we don't directly control). However we do not have zerotier installed on the guests (for cost reasons), only the hosts.

I'm always on the hunt for GUI based tools for managing these devices, because we have a lot of routine troubleshooting that needs to be done by less experienced techs. GUI access to filesystems in particular is really helpful. I've played around with 45Drives navigator plugin for cockpit, over a tunnelled SSH connection, and that worked really well, but it's a hassle to set up. If that sort of thing could be automated down to a single click (presumably with some sort of config file that I could share with people who need access) that would be really cool.

[–] [email protected] 2 points 1 year ago (1 children)

Now I'm not familiar with the details of zerotier, but XPipe does support using other connections as gateways. So if your network is only accessible from the outside through one login/firewall server then you can connect to that server first and use it as a gateway for your second ssh connection to some server inside your network.

This functionality is technically completely separate from the SSH tunneling with port forwarding, but it does suffice in many cases. If you require proper SSH tunneling functionality, then that can for sure be added.

[–] [email protected] 1 points 1 year ago (1 children)

OK, yeah, that makes sense. I'll have a play around with it and see if it fits the bill.

(zerotier is basically just a device to device VPN. It works very much like Tailscale)

[–] [email protected] 1 points 1 year ago (1 children)

OK, I finally had a chance to sit down with this thing, and I'm really impressed. This is a very cool idea, and a really slick piece of software.

I ran into a fair errors, unfortunately, but I'll chalk that up to early days.

The only thing I can't seem to figure is if there's any way to download and upload files. That would be really handy, especially if it worked across multiple hops, like the scenario I was describing. Being able to directly pull a file from a docker container on a kvm guest that I'm connecting to via a connecting on the host would be awesome.

[–] [email protected] 1 points 1 year ago (1 children)

Yeah the initial woes should be ironed out fairly quickly. Each user runs this in a completely different environment, so it was to be expected that I didn't account and prepare for some cases.

In the file manager, the main way of uploading files is to just drag and drop the files into the current directory, either from your local desktop environment or from another open tab in XPipe. For downloads, there should be a box on the bottom left where you can drag and drop remote files into. Once dropped into it, they are downloaded, and once it's finished, you can then drag them into your local desktop environment just as you can do with local files. I guess this process can be made more intuitive though, I will try to figure things out.

[–] [email protected] 1 points 1 year ago (1 children)

Yeah, I spotted the download box, but when I tabbed over to my local environment it disappeared. Since I was downloading from the guest VM maybe that was one jump too many. It did show when I tabbed over to the host, but dragging the files out didn't seem to do anything.

[–] [email protected] 1 points 1 year ago (1 children)

I should be able to fix the download box not staying there when switching to the local environment. If you want to drag and drop files from one tab into another, you can drag the files, hover them over the tab you want to switch to, and then drop them there.

In theory you should be able to just drag the files out of the download box as normal files, so you should be able to drag them into the XPipe browser but also other applications running on your desktop. What are your system details so maybe I can try to reproduce that?

[–] [email protected] 1 points 1 year ago (1 children)

I was trying this out on Windows 10, using the portable version.

[–] [email protected] 1 points 1 year ago (1 children)

Alright, I will try to look into that and also try to make the interface and work flows more intuitive

[–] [email protected] 2 points 1 year ago (1 children)

I'll definitely be keeping an eye on this. Could be very useful for some of our workflows.

[–] [email protected] 1 points 1 year ago

An update: The latest release now has full support for SSH tunnels and I reworked the downloading and transfer process in the file browser.

[–] [email protected] 1 points 1 year ago (1 children)

Just downloaded this and tried it out on a Debian VPS I have. Ran into a bunch of bugs to the point I couldn’t really do anything with it, but I can see a bunch of potential in the UI. I really like the idea of being able to see an overview of shell, containers, files, etc. I have a bunch of self hosted Proxmox VMs and various VPSs I use on a daily basis, and whole I’m totally comfortable with the command line, this tool seems genuinely useful.

It seems like you have a bunch of functionality and UI implemented already, so I think taking a few weeks to just bug hunt would be super beneficial at this point. I’ll open up some GitHub issues when I have a minute later, but I ran into so many bugs in just 5 min that it was basically unusable which is extra frustrating because it really seems like it can be a useful tool if it works.

[–] [email protected] 1 points 1 year ago (1 children)

That is unfortunate that you had to deal with these bugs. The challenge here is that every setup and shell environment people run this in differs and there was only so much testing I was able to do on my end. Reporting bugs is very helpful to me and they can usually get fixed pretty quickly.

[–] [email protected] 1 points 1 year ago (1 children)

Just reopened the app and tried it again and figured out what happened. I had not entered a password in settings when adding the server since I connect using an ssh key. It detected I had docker but when I tried to click it, it errored out. If I had read the error, I would have seen that the problem was needing the password for sudo. I added the password to the server settings and now it's working.

I guess then the only real "bug" I found so far is that on macOS the app defaults to using iTerm2.app which is a 3rd party terminal app which I don't have installed, so I had to change it to Terminal.app. I know iTerm2 is popular, but I think the default should be the one everyone has installed, and let iTerm2 users select their app in settings, not the other way around. But that's more a UI/UX/onboarding experience thing than a real bug (though maybe it's possible to detect if iTerm2 is installed).

Anyway, I'm going to keep playing with this and will report anything I find. So far my second impression is that it just overall feels kind of sluggish and doesn't have the best UI feedback when you're waiting for things so I ended up clicking things more than once not thinking it was working then it would open multiple times (like clicking the root file directory).

Hope to see you keep working on this, it seems like a really cool idea.

[–] [email protected] 2 points 1 year ago (1 children)

That is good to hear it works at least to some degree now. In theory, xpipe does detect your installed terminals at launch and will select the most appropriate (i.e. prefer iTerm2 over Terminal.app if iTerm2 is installed), however it seems like there are some false positives as is it quite difficult to properly check whether an app is installed somewhere on macOS.

The UI feedback is definitely on my TODO list, having a status bar for example that tells you exactly what is currently going on would be very nice to have instead of just a spinner or nothing at all. I will see what I can do there

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

Awesome, that seems like a great idea. Since as I understand it, the app is essentially just running terminal commands, I think showing the currently running command would be a huge UX improvement. It would help both with knowing what's going on and with debugging any issues with the commands.

Right now I'm traveling and my home VPN connection isn't working for some reason, so I don't have access to most of the VMs I usually use daily, but as soon as I get access again I'll get them all added and really give this a proper test drive. I'll report any issues I run across or UX suggestions I can think of. It's great to see how well you take feedback!

Also funny enough, just due to talking about iTerm2, I went and downloaded it and found out about the split panes feature and I think I may now be a convert haha.

[–] [email protected] 2 points 1 year ago (1 children)

Yeah it could definitely show the currently running command, although it will be tricky to not spam the user with too many commands and information in short succession, but we will see.

I guess the main reason for the popularity of iTerm2 is that the normal Terminal.app is just a little bit too basic in terms of its features. It works fine but nowadays people expect a little bit more.

[–] [email protected] 1 points 1 year ago

Hmm yeah that’s a good point about spamming commands. Great example of why UI/UX is so hard…it’s easy to throw out suggestions that sound good but the devil is always in the details (and edge cases) ;)

load more comments
view more: next ›