Day 12 of 100 Days of Devops

Our monitoring tool has reported an issue in Stratos Datacenter. One of our app servers has an issue, as its Apache service is not reachable on port 8088 (which is the Apache port). The service itself could be down, the firewall could be at fault, or something else could be causing the issue.

Use tools like telnet, netstat, etc. to find and fix the issue. Also make sure Apache is reachable from the jump host without compromising any security settings.

Once fixed, you can test the same using command curl http://stapp01:8088 command from jump host.

Note: Please do not try to alter the existing index.html code, as it will lead to task failure.

Mind Mapping

As it mentioned in problem the application is crashed.But the question has mentioned the “Apache Service” so I thought may be I can look into Apache Service .Before that I would like to check the server is responding for specific port form jump host.

Telnet for initial port checking from remote

telnet stapp01 8088 it says connection reset by peers. That means the app server does not allow or listen the port. So we can think of the app server itself have some issue.

Geting into server as usual and I want to know what are the existing services running.And from so on I can trace back upwards.

What is running inside Appserver

sudo systemctl list-units --type=service Listing all the services running.

 UNIT                      LOAD   ACTIVE       SUB           JOB  DESCRIPTION      
    
  dbus.service              loaded active       running            D-Bus System Message B
us
● httpd.service             loaded failed       failed     
        The Apache HTTP Server
  iptables.service          loaded active       exited             IPv4 firewall with ipt
ables
  selinux-autorelabel-mark.service loaded active       exited             Mark the need t
o relabel after reboot
  sendmail.service          loaded active       running            Sendmail Mail Transpor
t Agent
  sm-client.service         loaded active       running            Sendmail Mail Transpor
t Client
  sshd.service              loaded active       running            OpenSSH server daemon
  systemd-journald.service  loaded active       running            Journal Service      
  systemd-logind.service    loaded active       running            Login Service        
  systemd-tmpfiles-setup.service loaded active       exited             Create Volatile F
iles and Directories
  user-runtime-dir@0.service loaded active       exited             User runtime director
y /run/user/0
  user@0.service            loaded deactivating final-sigkill stop User Manager for UID 0

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
JOB    = Pending job for the unit.

As you can see this service it self is failing.

httpd.service loaded failed failed

We can do some journal check for more details, you may use systemctl status httpd.service for summary . Then I found out the the there was port conflict.

[tony@stapp01 ~]$ sudo systemctl status httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabl
ed)
   Active: failed (Result: exit-code) since Sat 2025-12-06 15:16:15 UTC; 13min
 ago
     Docs: man:httpd.service(8)
  Process: 541 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, stat
us=1/FAILURE)
 Main PID: 541 (code=exited, status=1/FAILURE)
   Status: "Reading configuration..."

Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com httpd[541]: (98)Address already in use: A
H00072: make_sock: could not bind to address 0.0.0.0:8088
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com httpd[541]: no listening sockets availabl
e, shutting down
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com httpd[541]: AH00015: Unable to open logs
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com systemd[1]: httpd
.service: Child 541 belongs to httpd.service.
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com systemd[1]: httpd
.service: Main process exited, code=exited, status=1/FAILURE
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com systemd[1]: httpd
.service: Failed with result 'exit-code'.
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com systemd[1]: httpd
.service: Changed start -> failed
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com systemd[1]: httpd
.service: Job httpd.service/start finished, result=failed
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com systemd[1]: Faile
d to start The Apache HTTP Server.
Dec 06 15:16:15 stapp01.stratos.xfusioncorp.com systemd[1]: httpd
.service: Unit entered failed state.

Narrowing down to port

So we have to find port bindings and process on the machine that causes conflict. For that we will be using netstat.

 netstat -ano | grep ":8088"
tcp        0      0 127.0.0.1:8088          0.0.0.0:*               LISTEN      off (0.00/0/0)

And I found out the process ID is running nunder sendmail service. In production environment we will be configuring the sendmail port to change to another port. However our case here is not a big deal if we kill the sendmail.

so kill the sendmail process.

then now we can restart the httpd.service and check the status again. The status should have active and running.

After that we can check whether we have our traffic able to route to local app server.

telnet localhost 8088 it shows connected .

and still I cannot telnet from jump host. So this time we can looking into IPTables (Filrewall).

[tony@stapp01 ~]$ iptables -L
iptables v1.8.4 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)

[tony@stapp01 ~]$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination     

As always it was not allowed so we can allow the port access by modifyig the IP Tables.

[tony@stapp01 ~]$ sudo iptables -I INPUT 1 -p tcp --dport 8088 -j ACCEPT


[tony@stapp01 ~]$ sudo service iptables save