Day 14 of 100 Days of Devops

The production support team of xFusionCorp Industries has deployed some of the latest monitoring tools to keep an eye on every service, application, etc. running on the systems. One of the monitoring systems reported about Apache service unavailability on one of the app servers in Stratos DC.

Identify the faulty app host and fix the issue. Make sure Apache service is up and running on all app hosts. They might not have hosted any code yet on these servers, so you don’t need to worry if Apache isn’t serving any pages. Just make sure the service is up and running. Also, make sure Apache is running on port 8083 on all app servers.

Debugging Traces

Since we know the port , we scan the individual port is opened by using telnet

Apparently in the below logs stapp01 is refusing the connection

thor@jumphost ~$ telnet stapp01 8083
Trying 172.16.238.10...
telnet: connect to address 172.16.238.10: Connection refused
thor@jumphost ~$ telnet stapp02 8083
Trying 172.16.238.11...
Connected to stapp02.
Escape character is '^]'.
^C^]
telnet> ^CConnection closed.
thor@jumphost ~$ telnet stapp03 8083
Trying 172.16.238.12...
Connected to stapp03.
Escape character is '^]'.
^]
telnet> Connection closed.

So we get into the server and try to find out whats going on. So we did a quick check using

[tony@stapp01 ~]$ systemctl status httpd.service -l 

then we found out that we got port binding issue since the port was allocated by some process.

Who is allocating Apache Port

Its time to find who is allocating the apache port using netstat.

[tony@stapp01 ~]$ sudo  netstat -tulnpa | grep ":8083"
tcp        0      0 127.0.0.1:8083          0.0.0.0:*               LISTEN      778/sendmail: accep 

Resolution

Then we found out sendmail is using the port that we are looking for at PDI 778

Then we kill the process and restart the http.service

Check the service

[tony@stapp01 ~]$ systemctl status httpd.service -l
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2025-12-07 13:46:38 UTC; 21s ago 
     Docs: man:httpd(8)
           man:apachectl(8)
  Process: 925 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
 Main PID: 966 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /docker/a60c4139912a3d6bacc580e00ed6dda337fc93c720b5418cf145628b6cf76bf2/system.slice/httpd.service
           ├─966 /usr/sbin/httpd -DFOREGROUND
           ├─967 /usr/sbin/httpd -DFOREGROUND
           ├─968 /usr/sbin/httpd -DFOREGROUND
           ├─969 /usr/sbin/httpd -DFOREGROUND
           ├─970 /usr/sbin/httpd -DFOREGROUND
           └─971 /usr/sbin/httpd -DFOREGROUND

Dec 07 13:46:38 stapp01.stratos.xfusioncorp.com systemd[1]: Starting The Apache HTTP Server...
Dec 07 13:46:38 stapp01.stratos.xfusioncorp.com httpd[966]: AH00558: httpd: Could not reliably determine t...age
Dec 07 13:46:38 stapp01.stratos.xfusioncorp.com systemd[1]: Started The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.

Verification

redo the telnet check on same port with same server then it is connected .

thor@jumphost ~$ telnet stapp01 8083
Trying 172.16.238.10...
Connected to stapp01.
Escape character is '^]'.

Thats all for today , Thx Bye !