EC2 Hacked

A couple of weeks ago, I gave Brandon access to my AWS EC2 for his Service Oriented Architecture course. I didn’t think much of it until today. I logged in and got a shock, I had a bill of $1,474.31. The data transfer section reflected that there was 7,759.453 GB worth of data transfer out of the free tier.

Screenshot

I called Brandon over and asked him if he knew of it. I immediately logged in and started looking at the logs. A couple of things were going through my head at this point.

Was there a run away process that caused the data transfer?
Did a hacker manage to brute force my AWS password?
SSH password login was disabled so that’s not possible to brute force. (Even so, I have fail2ban set to deny access.)
Did Brandon create a new account that had low security?

I checked the AWS usage reports and graphed the data. Wow. There was certainly a spike in usage on the 13th.

Bandwidth Graph

I checked that there wasn’t any runaway process that were sending traffic. There wasn’t any new accounts that were created as well. I noticed Brandon installed Apache Tomcat so I went to look at the logs.

I found that there were brute force login attemps on his service. Tomcat was running on port 8080, a common port for development.

...
...
Sep 10, 2013 5:52:47 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "admin"
Sep 10, 2013 5:52:47 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "admin"
Sep 10, 2013 5:52:47 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "admin"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "tomcat"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "tomcat"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "root"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "root"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "root"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "manager"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "manager"
Sep 10, 2013 5:52:48 PM org.apache.catalina.realm.LockOutRealm authenticate
WARNING: An attempt was made to authenticate the locked user "manager"
Sep 10, 2013 5:59:39 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /var/lib/tomcat7/webapps/huzi.war
...
...

That was quick! Someone managed to deploy huzi.war, about 7 mins after the first brute force attempt. I asked Brandon if he changed his password to the admin interface. Turns out he left it as the default tomcat/tomcat combination.

Upon further inspection, I noticed that the logs on the 14th September was missing. Looking at the access logs, it wasn’t just one bot trying to brute force the service, there were multiple attempts made by different people. Some were still unsuccessful and kept trying until today.

I found that there were 3 different war applications that were deployed. huzi.war, asd.war and Memory.war. They were of different origins but Memory.war did the damage associated with the cost and bandwidth graphs.

huzi.war logs:

catalina.2013-09-10.log

...
...
Sep 10, 2013 5:59:39 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /var/lib/tomcat7/webapps/huzi.war
...
...

localhost_access_log.2013-09-10

...
...
112.65.211.122 - - [10/Sep/2013:17:59:28 +0000] "GET /manager/html HTTP/1.1" 401 2486
112.65.211.122 - tomcat [10/Sep/2013:17:59:30 +0000] "GET /manager/html HTTP/1.1" 200 14107
112.65.211.122 - tomcat [10/Sep/2013:17:59:31 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 200 2066
112.65.211.122 - tomcat [10/Sep/2013:17:59:31 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 200 7279
112.65.211.122 - - [10/Sep/2013:17:59:31 +0000] "GET /favicon.ico HTTP/1.1" 404 988
112.65.211.122 - tomcat [10/Sep/2013:17:59:40 +0000] "POST /manager/html/upload?org.apache.catalina.filters.CSRF_NONCE=4320A9351047FF7D39C2008FEE8DED1A HTTP/1.1" 200 15782
112.65.211.122 - - [10/Sep/2013:17:59:40 +0000] "GET /favicon.ico HTTP/1.1" 404 988
112.65.211.122 - - [10/Sep/2013:17:59:42 +0000] "GET /huzi HTTP/1.1" 302 -
112.65.211.122 - - [10/Sep/2013:17:59:44 +0000] "GET /huzi/ HTTP/1.1" 200 3319
112.65.211.122 - - [10/Sep/2013:17:59:46 +0000] "POST /huzi/ HTTP/1.1" 200 6571
...
...

asd.war logs:

catalina.2013-09-13.log

...
...
Sep 13, 2013 1:49:17 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /var/lib/tomcat7/webapps/asd.war
...
...

localhost_access_log.2013-09-13

...
...
172.240.97.226 - tomcat [13/Sep/2013:13:49:08 +0000] "GET /manager/html HTTP/1.1" 200 15796
172.240.97.226 - tomcat [13/Sep/2013:13:49:09 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 200 7279
172.240.97.226 - tomcat [13/Sep/2013:13:49:09 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 200 2066
172.240.97.226 - tomcat [13/Sep/2013:13:49:17 +0000] "POST /manager/html/upload?org.apache.catalina.filters.CSRF_NONCE=4F47618AB5CEBCBD1A5C48780752C12D HTTP/1.1" 200 17471
172.240.97.226 - - [13/Sep/2013:13:49:20 +0000] "GET /asd HTTP/1.1" 302 -
172.240.97.226 - - [13/Sep/2013:13:49:28 +0000] "GET /asd/ HTTP/1.1" 200 7853
172.240.97.226 - - [13/Sep/2013:13:49:28 +0000] "GET /asd/?Javascript HTTP/1.1" 200 3849
199.59.160.152 - - [13/Sep/2013:13:50:12 +0000] "GET /asd/ HTTP/1.1" 200 7853
199.59.160.152 - - [13/Sep/2013:13:50:13 +0000] "GET /asd/?Javascript HTTP/1.1" 200 3849
199.59.160.152 - - [13/Sep/2013:13:50:14 +0000] "GET /asd/?sort=1&dir=%2Fvar%2Flib%2Ftomcat7%2Fwebapps HTTP/1.1" 200 10238
199.59.160.152 - - [13/Sep/2013:13:50:23 +0000] "GET /asd/?first&uplMonitor=C%3A%5Ch HTTP/1.1" 200 856
199.59.160.152 - - [13/Sep/2013:13:50:25 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 865
199.59.160.152 - - [13/Sep/2013:13:50:27 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:30 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 868
199.59.160.152 - - [13/Sep/2013:13:50:32 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:34 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:36 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:39 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:41 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:43 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:45 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 866
199.59.160.152 - - [13/Sep/2013:13:50:47 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 864
199.59.160.152 - - [13/Sep/2013:13:50:50 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 864
199.59.160.152 - - [13/Sep/2013:13:50:53 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 864
199.59.160.152 - - [13/Sep/2013:13:50:55 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 864
199.59.160.152 - - [13/Sep/2013:13:50:58 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 863
199.59.160.152 - - [13/Sep/2013:13:51:00 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 863
199.59.160.152 - - [13/Sep/2013:13:51:02 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 863
199.59.160.152 - - [13/Sep/2013:13:51:03 +0000] "POST /asd/ HTTP/1.1" 200 11078
199.59.160.152 - - [13/Sep/2013:13:51:05 +0000] "GET /asd/?uplMonitor=C%3A%5Ch HTTP/1.1" 200 579
199.59.160.152 - - [13/Sep/2013:13:51:14 +0000] "POST /asd/ HTTP/1.1" 200 2277
199.59.160.152 - - [13/Sep/2013:13:51:26 +0000] "POST /asd/ HTTP/1.1" 200 2277
199.59.160.152 - - [13/Sep/2013:13:51:30 +0000] "POST /asd/ HTTP/1.1" 200 2277
...
...

Memory.war logs:

catalina.2013-09-11.log

...
...
Sep 11, 2013 7:11:39 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /var/lib/tomcat7/webapps/Memory.war
Sep 11, 2013 7:12:13 AM org.apache.catalina.startup.HostConfig checkResources
INFO: Undeploying context [/axis2]
...
...

localhost_access_log.2013-09-11

...
...
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - tomcat [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 200 18946
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:04:00:08 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - - [11/Sep/2013:07:11:26 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - tomcat [11/Sep/2013:07:11:31 +0000] "GET /manager/html HTTP/1.1" 200 17449
192.210.52.72 - tomcat [11/Sep/2013:07:11:31 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 200 7279
192.210.52.72 - tomcat [11/Sep/2013:07:11:31 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 200 2066
192.210.52.72 - - [11/Sep/2013:07:11:31 +0000] "GET /favicon.ico HTTP/1.1" 404 988
192.210.52.72 - tomcat [11/Sep/2013:07:11:40 +0000] "POST /manager/html/upload?org.apache.catalina.filters.CSRF_NONCE=98A6825304BB265C4851C3E0558E1A02 HTTP/1.1" 200 19138
192.210.52.72 - - [11/Sep/2013:07:11:40 +0000] "GET /favicon.ico HTTP/1.1" 404 988
192.210.52.72 - - [11/Sep/2013:07:11:44 +0000] "GET /Memory HTTP/1.1" 302 -
192.210.52.72 - - [11/Sep/2013:07:11:45 +0000] "GET /Memory/ HTTP/1.1" 200 1103
192.210.52.72 - - [11/Sep/2013:07:11:51 +0000] "POST /Memory/ HTTP/1.1" 200 6170
192.210.52.72 - - [11/Sep/2013:07:11:54 +0000] "GET /Memory/?action=filesystem HTTP/1.1" 200 6826
192.210.52.72 - - [11/Sep/2013:07:11:56 +0000] "GET /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/ HTTP/1.1" 200 9272
192.210.52.72 - - [11/Sep/2013:07:12:04 +0000] "POST /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/&fsAction=deleteFile HTTP/1.1" 200 2268
192.210.52.72 - - [11/Sep/2013:07:12:06 +0000] "GET /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/ HTTP/1.1" 200 8046
192.210.52.72 - - [11/Sep/2013:07:12:11 +0000] "GET /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/axis2/ HTTP/1.1" 200 7294
192.210.52.72 - - [11/Sep/2013:07:12:13 +0000] "GET /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/axis2/WEB-INF/ HTTP/1.1" 200 8470
192.210.52.72 - - [11/Sep/2013:07:12:17 +0000] "POST /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/axis2/WEB-INF/&fsAction=deleteFile HTTP/1.1" 200 2231
...
...

localhost_access_log.2013-09-13

...
...
192.210.52.72 - - [13/Sep/2013:22:41:29 +0000] "GET /manager/html HTTP/1.1" 401 2486
192.210.52.72 - tomcat [13/Sep/2013:22:41:36 +0000] "GET /manager/html HTTP/1.1" 200 17471
192.210.52.72 - tomcat [13/Sep/2013:22:41:37 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 200 7279
192.210.52.72 - tomcat [13/Sep/2013:22:41:37 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 200 2066
192.210.52.72 - - [13/Sep/2013:22:41:37 +0000] "GET /favicon.ico HTTP/1.1" 404 988
192.210.52.72 - - [13/Sep/2013:22:41:43 +0000] "GET /Memory HTTP/1.1" 302 -
192.210.52.72 - - [13/Sep/2013:22:41:43 +0000] "GET /Memory/ HTTP/1.1" 200 1103
192.210.52.72 - - [13/Sep/2013:22:41:43 +0000] "GET /favicon.ico HTTP/1.1" 404 988
192.210.52.72 - - [13/Sep/2013:22:41:49 +0000] "POST /Memory/ HTTP/1.1" 200 6170
192.210.52.72 - - [13/Sep/2013:22:42:10 +0000] "POST /Memory/ HTTP/1.1" 200 6170
192.210.52.72 - - [13/Sep/2013:22:42:13 +0000] "GET /Memory/?action=command HTTP/1.1" 200 2277
192.210.52.72 - - [13/Sep/2013:22:42:20 +0000] "POST /Memory/?action=command HTTP/1.1" 200 5406
192.210.52.72 - - [13/Sep/2013:22:42:29 +0000] "POST /Memory/?action=command HTTP/1.1" 200 2289
192.210.52.72 - - [13/Sep/2013:22:42:59 +0000] "POST /Memory/?action=command HTTP/1.1" 200 2289
49.90.29.5 - - [13/Sep/2013:22:43:03 +0000] "GET /manager/html HTTP/1.1" 401 2486
49.90.29.5 - tomcat [13/Sep/2013:22:43:15 +0000] "GET /manager/html HTTP/1.1" 200 17471
192.210.52.72 - - [13/Sep/2013:22:43:23 +0000] "POST /Memory/?action=command HTTP/1.1" 200 2286
49.90.29.5 - tomcat [13/Sep/2013:22:43:29 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 200 2066
49.90.29.5 - tomcat [13/Sep/2013:22:43:29 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 200 7279
192.210.52.72 - - [13/Sep/2013:22:43:30 +0000] "POST /Memory/?action=command HTTP/1.1" 200 5361
192.210.52.72 - - [13/Sep/2013:22:43:39 +0000] "GET /Memory/?action=filesystem HTTP/1.1" 200 6826
192.210.52.72 - - [13/Sep/2013:22:43:41 +0000] "GET /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/ HTTP/1.1" 200 8878
192.210.52.72 - - [13/Sep/2013:22:43:47 +0000] "POST /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/&fsAction=deleteFile HTTP/1.1" 200 2268
192.210.52.72 - - [13/Sep/2013:22:43:49 +0000] "GET /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/ HTTP/1.1" 200 7572
192.210.52.72 - - [13/Sep/2013:22:43:58 +0000] "GET /Memory/7.jsp HTTP/1.1" 200 9726
192.210.52.72 - - [13/Sep/2013:22:44:04 +0000] "GET /Memory/7.jsp?first&uplMonitor=C%3A%5Cfakepath%5Ctomcat-jy HTTP/1.1" 200 1827
192.210.52.72 - - [13/Sep/2013:22:44:07 +0000] "GET /Memory/7.jsp?uplMonitor=C%3A%5Cfakepath%5Ctomcat-jy HTTP/1.1" 200 1838
192.210.52.72 - - [13/Sep/2013:22:44:09 +0000] "GET /Memory/7.jsp?uplMonitor=C%3A%5Cfakepath%5Ctomcat-jy HTTP/1.1" 200 1836
192.210.52.72 - - [13/Sep/2013:22:44:10 +0000] "POST /Memory/7.jsp HTTP/1.1" 200 10526
192.210.52.72 - - [13/Sep/2013:22:44:11 +0000] "GET /Memory/7.jsp?uplMonitor=C%3A%5Cfakepath%5Ctomcat-jy HTTP/1.1" 200 1523
192.210.52.72 - - [13/Sep/2013:22:44:17 +0000] "GET /Memory/ HTTP/1.1" 200 6170
192.210.52.72 - - [13/Sep/2013:22:44:19 +0000] "GET /Memory/?action=filesystem HTTP/1.1" 200 7351
192.210.52.72 - - [13/Sep/2013:22:44:38 +0000] "GET /Memory/?action=command HTTP/1.1" 200 2277
192.210.52.72 - - [13/Sep/2013:22:44:40 +0000] "POST /Memory/?action=command HTTP/1.1" 200 2329
49.90.29.5 - - [13/Sep/2013:22:44:43 +0000] "GET /favicon.ico HTTP/1.1" 404 988
192.210.52.72 - - [13/Sep/2013:22:44:49 +0000] "POST /Memory/?action=command HTTP/1.1" 200 2343
192.210.52.72 - - [13/Sep/2013:22:45:01 +0000] "GET /Memory/?action=filesystem HTTP/1.1" 200 7351
192.210.52.72 - - [13/Sep/2013:22:45:02 +0000] "GET /Memory/?action=filesystem&curPath=/var/lib/tomcat7/webapps/ HTTP/1.1" 200 6604
192.210.52.72 - - [13/Sep/2013:22:45:07 +0000] "GET /Memory/?action=command HTTP/1.1" 200 2277
192.210.52.72 - - [13/Sep/2013:22:45:17 +0000] "POST /Memory/?action=command HTTP/1.1" 200 2312
192.210.52.72 - - [13/Sep/2013:22:45:18 +0000] "POST /Memory/?action=command HTTP/1.1" 404 -
49.90.29.5 - tomcat [13/Sep/2013:22:45:36 +0000] "GET /manager/html HTTP/1.1" 200 10770
49.90.29.5 - tomcat [13/Sep/2013:22:45:36 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 304 -
49.90.29.5 - tomcat [13/Sep/2013:22:45:38 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 304 -
49.90.29.5 - - [13/Sep/2013:22:45:38 +0000] "GET /favicon.ico HTTP/1.1" 404 -
192.210.52.72 - tomcat [13/Sep/2013:22:45:47 +0000] "GET /manager/html HTTP/1.1" 200 10770
192.210.52.72 - tomcat [13/Sep/2013:22:45:47 +0000] "GET /manager/images/tomcat.gif HTTP/1.1" 304 -
192.210.52.72 - tomcat [13/Sep/2013:22:45:47 +0000] "GET /manager/images/asf-logo.gif HTTP/1.1" 304 -
192.210.52.72 - tomcat [13/Sep/2013:22:45:54 +0000] "POST /manager/html/upload?org.apache.catalina.filters.CSRF_NONCE=461F1222078E9A0B2DF7C7E5DDEE958C HTTP/1.1" 200 10894
...
...

We can see that the service definitely got compromised. Memory.war was the only application left behind together with these logs.

What actually happened on the 14th, we would never know. Brandon reported this incident and submitted my findings to his lecturer and AWS.

Lessons learnt

Lecturers should advice proper security practices such as renaming your Tomcat admin account and changing the default password to prevent these brute forcing bots from actually getting in.
Brandon should have been slightly security concious about the setup of such services. Changing default passwords should be on the top of everyones head when deploying a service that’s exposed to the entire world.
I should have checked what Brandon was doing with my server and adviced him against bad security practices.
I should have set up AWS billing alerts to prevent the bill shock.

Feel free to contact me for access to the entire package.

Update

It was the first time the lecturer used AWS as a platform to teach Service Oriented Architecture. I thought that it was actually more realistic to deploy services on cloud providers. Turns out that 2 other students were compromised as well and the lecturer decided not to use AWS for the next session. Why?! This clearly shows that students are not security concious and should be taught how to properly secure and deploy services that were exposed to other malicious parties.

Published on 22 Sep 2013 by Stanley Tan

Home