TCP echo server

[danielsz]

It's great to read your blog, Matheus!
Maybe a few examples would be nice to have. A TCP echo server makes it absolutely clear to anyone wondering that it is possible to write interesting programs with only system calls on linux. In particular, I had small web (smolweb) enthusiasts in mind. This simple program can serve as basis to implement protocols like Gemini, Spartan or Gopher.
Note: Gemini would need to be proxied over TLS in order to comply with the protocol.

Usage:

• Run the server (in lone)
• Client: nc localhost 8080.
• Anything you type will be echoed

[matheusmoreira]

Nice to see you again! This is probably the first real program written in lone lisp. All the programs I've been writing until now have been test cases. This is huge!!

Just tested it on master and it correctly handled twenty thousand parallel clients!! Huge latency spikes due to the listen backlog, bumping that improves handling of bursty arrivals but at some point it hits the single threaded loop wall.

I see you've built the socket structure manually with the bytes primitives. A structure module has been on my TODO list since forever. Hopefully users won't have to do this in the future. Still, the code is surprisingly clean despite the lack of it. Should be even cleaner once structure lands!

I'm very much aligned with the small web too, would be very happy if folks in this space found a use for lone! A production server would likely want to use io_uring or epoll here. My long term goal for lone is Erlang/BEAM style lightweight processes, that could also be a viable implementation strategy once it lands. I've benchmarked lone's stacks for simple programs at less than 1 KiB each, that should provide for hundreds of thousands to millions of lisp threads.

Lone uses the .ln file extension, would you mind renaming the file?

[matheusmoreira]

It's a great example program! Made some small suggestions.

[matheusmoreira]

It's possible to use write-u16 here.

(write-u16 sockaddr 0 AF_INET); sin_family, host byte order

Should also work correctly on big endian targets.

[matheusmoreira]

It's possible to use write-u16be here.

(write-u16be sockaddr 2 8080); sin_port, network byte order

[danielsz]

OK, done!

Thanks for the comments and insights. I am very curious about structure. I will be happy to follow up on that. I also like the idea of exploring io_uring and/or epoll based solutions.

[matheusmoreira]

Thanks, it's looking good!

It seems the rename duplicated the file instead: example/tcp-echo is still present alongside example/tcp-echo.ln. It should be deleted.

It's OK to force push the branch with a cleaned up history too.

The failing CI check seems to be just GitHub flakiness, not a blocker. With the above changes, I should be able to merge your work with full attribution.

[matheusmoreira]

The structure module I envision would work lile this.

struct sockaddr_in {
    short  sin_family;
    __be16 sin_port;
    __be32 sin_addr;
};

(structure sockaddr_in
  (sin_family short)
  (sin_port   u16be)
  (sin_addr   u32be))

; automatically calculates sizes,
; alignments and offsets
; generates readers and writers
; that work on bytes structures

(sockaddr_in/sin_family-read buffer)
(sockaddr_in/sin_port-read   buffer)
(sockaddr_in/sin_addr-read   buffer)

(sockaddr_in/sin_family-write buffer family)
(sockaddr_in/sin_port-write   buffer port)
(sockaddr_in/sin_addr-write   buffer address)

I investigated io_uring and it's currently impossible to use it. Lone needs the ability to point a bytes object at an arbitrary memory address. It's on the TODO list! I think epoll might already be working though, I'll test it.

[danielsz]

Forced push the whole thing under one commit. If there's anything else, please let me know. Thank you.

[matheusmoreira]

Awesome! Ready to merge!