How to use proxy on Linux

This tutorial is targeted at readers who have acquired a working proxy (in this article referring to one that is used to access blocked websites like huggingface.co) with a beginner-friendly graphical interface on their own Windows/MacOS computer, but have just been given access to a Linux server. Thus it will not cover advanced networking topics.

You may generalize the practices to use proxy on WSL or command line applications on your own computer.

0. What is a Proxy?

When your computer cannot access a website directly, a proxy server can help you transfer the request to the website server and send the response back to you. A proxy client running your computer helps communicate with the proxy server. Other programs on your computer, like the browser and command line applications, need to be aware of the proxy client in order to use it.

Using a mirror site can cover many scenarios without using a proxy.

1. Test if a proxy is needed

Run curl https://www.huggingface.co. If you see the HTML content of the website instead of an error message, you don’t need additional configuration. This may be because the server is abroad, or someone has already set up a proxy.

Do not use ping.

2. Install proxy client on Linux

In order to do this you first need to learn to use the command line interface version of your proxy client, often referred to as the core in the graphical interface. You need to download and run the binary (executable file) of the client and write the configuration file for it. After testing, you can upload the binary (may be different from that on your own computer due to architecture and platform differences) and the configuration file to the Linux server. Optionally, you can make the proxy client run in the background on statup.

The exact steps and concepts are too complicated for this article, but you can find the official documentation of the proxy client, working example, tutorials and other helpful materials by searching Google in Chinese. This is one of the few cases where you should search in Chinese.

Afterwards, you will know the address of the proxy client on your server. For example, http://localhost:1080. I will use this address in the rest of the article but you should replace it with the actual address.

Using the SOCKS5 protocol, i.e. socks5://localhost:1080, instead of the HTTP protocol, is not recommended because not all applications support it. Unless you know you have a reason to use SOCKS5, stick with HTTP.

If the server is shared by multiple users, you may scan the ports and potentially find other users’ proxy clients.

What is `localhost`?

In networking localhost means ‘the computer itself’. Ask GPT for more information.

3. Set up proxy for command line applications

What you need to do now is to get the applications to know about the presence of the proxy client. The graphical client on your own computer does that for you automatically, but on the server you have to do it manually.

For most applications

Most applications check the environment variables to find the proxy client. However, the exact variable name is notoriously inconsistent among applications. So you have to set all of them.

1
2

export http_proxy=http://localhost:1080; export https_proxy=$http_proxy; export ftp_proxy=$http_proxy; export all_proxy=$http_proxy; export HTTP_PROXY=$http_proxy; export HTTPS_PROXY=$http_proxy; export FTP_PROXY=$http_proxy; export ALL_PROXY=$http_proxy;
export no_proxy="127.0.0.1/8"; export NO_PROXY=$no_proxy

Information about environment variables, like PATH, LD_LIBRARY_PATH and those above, can be easily found in the Internet or by asking GPT.

You can use curl -v https://www.huggingface.co to test if the environment variables are present and correct.

Expected outcome

> curl -v https://www.huggingface.co
* Uses proxy env variable https_proxy == 'http://127.0.0.1:1080'
*   Trying 127.0.0.1:1080...
* Connected to 127.0.0.1 (127.0.0.1) port 1080
* CONNECT tunnel: HTTP/1.1 negotiated

# Other information omitted

<html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
</body>
</html>

Applications known to respect these environment variables include curl, wget, pip and git. Though some may have their own configuration files that can override the environment variables.

For `apt`

Using apt requires sudo, which doesn’t automatically inherit the environment variables in your current shell. You should run:

1	sudo -E apt update

The -E flag tells sudo to preserve the environment variables.

This also applies to package managers for other distributions, like yum or dnf.

For `docker`

The Docker Engine does respect environment variables, but only does so at the start-up of dockerd instead of the Docker CLI, docker. So sudo -E docker run ... won’t work.

What further complicates the matter is that there are two things requiring a proxy:

The Docker Engine itself, which needs to download images from the Docker Hub, and build images using packages from the Internet.
The Docker containers, which may need to download assets.

Even worse, the localhost of the container may be different from that of the host, meaning a different address is needed.

Therefore, the recommended way to tackle all these is to modify the Docker configuration file according to the official documentation (You need to read both links). When filling the proxy address, instead of localhost, use the hostname of the server, which can be found by running hostname.

Afterwards, Docker Engine will use the proxy to download the images, and set up proxy-related environment variables when building images and inside the containers.

Docker Desktop behaves totally differently from Docker Engine in this regard. Documentation for configuration is here. Containers in Docker Desktop are transparently proxied instead of using environment variables.

For `ssh`

ssh does not support using http proxy at all. You should not use this anyway because usually you can access the server directly or use a VPN.

If you are not satisfied with the quality of the VPN you use to connect to the server. You can forward the ssh port on the server to your own computer using tools like frp or, in some cases, your proxy client.

For `git`

git has its own configuration file which can override the environment variables.

git will only use the proxy when the URL of the repository starts with https://. If ssh is used, git will use the ssh client in the system and thus ignore the proxy.

Connecting to github.com over ssh is not blocked as of writing this article, eliminating the need for a proxy.

Despite this, uploading your private ssh key to a shared server is a security risk and you should not do that. To manage a private repository on a remote server, you can use personal access tokens instead of ssh keys

For `vscode-server`

VSCode uses command line tools in your system or the server’s system as much as possible. For example, system git is used by the Git extension, and system ssh is used by the Remote - SSH extension to connect to the server. Therefore, whether a proxy is used depends on the behavior of these tools. git, ssh and other tools used by extensions are subprocess of VSCode and thus inherit the environment variables in the launch context of VSCode.

VSCode also has its own proxy settings, which affects web requests made by the editor itself or extensions that directly access the internet, such as Github Copilot. This settings can be different for your local VSCode and the VSCode server.

4. Transparent proxy

Applications being aware of the proxy client on your computer have to change the way they access the Internet in order to use it. However, if a transparent proxy (also referred to as the TUN mode) is set up, applications can access the Internet as if no proxy is present, and the proxy client will intercept the requests and responses.

Although some graphical proxy clients achieves this within a single click on your computer, it’s not that straightforward on a server. A misconfigured transparent proxy can cause trouble because it usually affects all applications on the server. You can even lose access to the server! Therefore, this is not recommended unless thoroughly tested.

proxychains is a tool that transparently proxies a single application, which is safer than a system-wide transparent proxy. You may try it out if other methods don’t work.

Postscript

I dislike graphical proxy clients because they hide the details for you, which is what you need to learn as your skill level grow.