Codiusd times out / stops responding to request from Nginx

[jonaagenilsen]

2018/09/17 09:51:08 [error] 1470#0: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 213.154.230.246, server: , request: "GET /info HTTP/1.1", upstream: "http://127.0.0.1:3000/info", host: "hodling-xrp.no"

This can happen with services newly restarted (1-2 hours later). Or even earlier. Restarting Codiusd resolves the problem. Have eight hosts and monitoring https-requests to them all the time. This happens atleast 2-3 times each day (timeout).

Running v1.2.5 with latest version of "everything" on all hosts:

npm -g list --depth=0
├── codiusd@1.2.5
├── moneyd@4.1.1
├── moneyd-uplink-xrp@1.2.5
└── npm@6.4.1

yum list installed | grep hyper
hyper-container.x86_64               1.1.0-1.el7                       installed
hyperstart.x86_64                    1.1.0-1.el7                       installed
qemu-hyper.x86_64                    2.4.1-3.el7.centos                installed

CentOS Linux release 7.5.1804 (Core)
Linux hodling-xrp.no 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

grep -cw vmx /proc/cpuinfo
4

Tried tweaking Nginx. Timeouts still happening with Codiusd. Host is working normally and contract can be uploaded (when not in timeout-state)

[jonaagenilsen]

/etc/systemd/system/codiusd.service

[Unit]
Description=Codiusd
After=network.target nss-lookup.target
[Service]
#ExecStart=/usr/local/bin/codiusd
ExecStart=/usr/bin/npm start
WorkingDirectory=/usr/lib/node_modules/codiusd
Environment=DEBUG=*
Environment=CODIUS_PUBLIC_URI=https://hodling-xrp.no
Environment=CODIUS_XRP_PER_MONTH=10
Environment=CODIUS_MAX_MEMORY_FRACTION=0.97
Environment=CODIUS_ADDITIONAL_HOST_INFO=true
Environment=CODIUS_DISABLE_SELF_TEST=false
Environment=CODIUS_ADMIN_API=true
Environment=CODIUS_ADMIN_PORT=3001
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=codiusd
User=root
Group=root
[Install]
WantedBy=multi-user.target

[jonaagenilsen]

/etc/nginx/conf.d/codius.conf

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' $http_connection;
}

server {
    listen 443 ssl;

    ssl_certificate /etc/letsencrypt/live/hodling-xrp.no/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/hodling-xrp.no/privkey.pem;

    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/nginx/dhparam.pem;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
    ssl_ecdh_curve secp384r1;
    ssl_session_timeout 10m;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 1.1.1.1 1.0.0.1 valid=300s;
    resolver_timeout 5s;
    add_header Strict-Transport-Security 'max-age=63072000; includeSubDomains; preload';
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection '1; mode=block';

    location / {
      proxy_pass http://127.0.0.1:3000;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_buffering off;
      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;
      send_timeout               90;
    }
}

[jonaagenilsen]

Haven't done anything to Codius service last 16 hours:

[jonaagenilsen]

Normal response now after restart of codiusd service:

[jonaagenilsen]

An hour later.. things start to choke

[jonaagenilsen]

Correction:
Seems to only be the landingpage timing out (/).
Not /info /version .. Strange.

[jonaagenilsen]

More spam :)

/ stops working, /info working like a charm.

[jonaagenilsen]

With these changes:

nano /etc/systemd/system/codiusd.service
Environment="DEBUG=codiusd*,ilp*,-*Peer*,-*trace*"

It seems uploading stops when the following enters the logs

systemctl restart codiusd && journalctl -f -u codiusd

Sep 20 20:58:56 hodl-xrp.org codiusd[24096]: 2018-09-20T18:58:56.632Z ilp-plugin-mini-accounts:debug debug incoming ws closed. error= { Error: read EHOSTUNREACH
Sep 20 20:58:56 hodl-xrp.org codiusd[24096]: at TCP.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'EHOSTUNREACH', code: 'EHOSTUNREACH', syscall: 'read' }
Sep 20 20:58:56 hodl-xrp.org codiusd[24096]: 2018-09-20T18:58:56.633Z ilp-plugin-mini-accounts:debug debug incoming ws closed. error= 1006

Again, restarting Codiusd, everything works like a charm again.

[granuz88]

I have the same problem too!
After 1-2 hours the service times out.
I tried to reinstall from scratch on both Centos 7.5 and Ubuntu 16.04, the situation does not change

[PhilieB]

Also experiencing the same issue and have worked with Jon closley to resolve but doesnt seem to be a host issue. Both codius.itechnical.ie and codius.itechnical.ie/peers lag horribly after restarting the services within 1 hour. On the 24th, 25th and 26th the lag disappeared but now today it is back again. Any info would be apprecited on the issue

[wilsonianb]

Thanks for the info.
I'm trying to figure this out but have only managed to reproduce once in three days.

[jonaagenilsen]

granuz88 (above) recreated it building up from scratch. So for reproducing on a VM or similar, install

├── codiusd@1.2.5
├── moneyd@4.1.1
├── moneyd-uplink-xrp@1.2.5
└── npm@6.4.1

hyper-container.x86_64 1.1.0-1.el7 installed
hyperstart.x86_64 1.1.0-1.el7 installed

and use standard configs. Maybe snipp out configs pasted from me in those earlier posts? Same config most of us in https://gitter.im/codius/codius-chat are running :)

[granuz88]

In both cases I have used the codius installer script without modification:
https://github.com/xrp-community/codius-install

[wilsonianb]

Thanks
I'll try that script and then try some codius-chat configs if needed.

[t3silver]

Having the same issue myself, please let me know if you need to collect any data from my host in order to attempt to resolve this issue. My codius installation did not leverage the script.