A (possible) solution to COVID-19

Like nearly everyone on the planet I am worried about COVID-19. SARS-CoV-2 (the virus that causes COVID-19) appears to be killing between 1% to 3.5% of the people it infects and has a R0 (i.e. how many new people each person infected goes onto infect) of between 2.5 to 3.9. Left to run wild, the virus will likely kill tens of millions of people worldwide.

The governments of the world have implemented strict population isolation protocols to try and limit the spread of the virus, but the economic cost of this is extremely high. A vaccine for COVID-19 is 12 to 18 months away (at best).

We are stuck in a diabolic situation where the only way to prevent the economy sliding into a slump deeper than the Great Depression is to consign many tens of millions of people to an early grave. Is there a way out?

SARS-Cov-2 Viral Diversity

SARS-CoV-2 like all viruses mutates (changes) overtime. Many of these genetic changes are small (single nucleotides) that are not important to the replications or transmission of the virus from person to person, but they can be used to identify the origin of the virus. DeCODE genetics has been testing Icelanders for COVID-19 and genome sequencing the SARS-CoV-2 strains isolated. They have found two very important pieces of information:

  1. More than 50% of the people infected with SARS-CoV-19 are asymptomatic (i.e. they have no illness).
  2. They can identify the geographical origin of the strains by the genetic differences (mutations) between the different strains.

Furthermore, researchers in China have identified a mutant strain of SARS-CoV-2 which appears to be less pathogenic than most strains infecting people. This strain (ZJ01) had single nucleotide mutations in a key functional gene that made it less able to spread through the body.

This data suggests a simple and testable hypothesis – there are natural strains of SARS-CoV-2 in the world that have mutated to be non-pathogenic (asymptomatic), but are still infective and will provide immunity to the more pathogenic (deadly) strains.

If we can find one of these non-pathogenic viral strains out in the wild we could give it to everyone in the world and solve our diabolic problem. This non-pathogenic strain would act much like the live attenuated (oral) polio vaccine.

How do we find the attenuated SARS-CoV-2 strains?

This hypothesis is worthless if we have no way of finding any of these non-pathogenic SARS-CoV-2 viral strains. Luckily there is a quick and cheap way to find these strains if they exist – test asymptomatic people for COVID-19 and then genome sequence the SARS-CoV-2 strain that has infected them with the aim of identifying a virus with mutation(s) in an essential viral gene(s). This is what the Chinese researched did to find their less pathogenic strain

This approach is cheap (a couple of hundreds of dollars a virus strain) and quick (a week or less). With almost no effort or cost we could sequence a few thousand viral strains from asymptomatic people until we find a virus strain with the right mutations to make it harmless and which is in effect an attenuated vaccine. We would know that this strain can still reproduce in people and lead to immunity, but not make people seriously ill. 

What viral mutations are we looking for in a good non-pathogenic viral strain?

We would ideally be looking for a virus strain with a large(ish) deletion in an essential viral gene. This sort of mutation is easy to spot in the SARS-CoV-2 genome data, and because the genetic information has been removed, it makes the virus very unlikely to be able to mutate back into a dangerous strain. Ideally, the strain identified will have infected a number of other people in the local area too so we can know it is safe.

Has this ever been done before?

Yes. The polio, measles, rubella, mumps, and chicken pox vaccines are all live attenuated viruses. Even something as dangerous as smallpox was controlled in the 18th century using a variation of this idea called Variolation. The idea was the doctor would deliberately infect you with a less harmful strain of smallpox to make you immune to the more deadly strains of smallpox. Of course, they didn’t know how this approach worked in the 18th century, but it was still very effective and millions of people were saved from dying from smallpox by it.

Some people have been calling dosing with a low dose of the virus variolation, this is not what variolation is. While such a low-dose approach might make the COVID-19 less dangerous to the person being infected, it doesn’t make the virus any less dangerous for those around you that you might infect. Such an approach could also not be used on the vulnerable leaving them exposed to the illness.

What are the risks?

The major risk is the virus we think is safe is not 100% safe. While we can use community spread of the identified strain to estimate how safe it will be (i.e. if it has infected 1000 people and none have got seriously ill then we should have a pretty good idea that it is safe), but our knowledge will be incomplete. We can of course spend the next few years testing and trialling, but if we do this by the time any strain is shown to be 99.999% safe (not even the polio vaccine is 100% safe) we will have all got COVID-19 and the world’s economy will be a smoking ruin.

We have a choice of taking some small risk now, or face the certainty of a much worse problem later. Time to accept some risk and do something.

Update. Q & A

I have been getting a few questions on this post so I thought I would address them here.

How do you know there is an attenuated viral strain out there?

Because such strains have already been found. I am hypothesising that there is more than one based on the known mutation rate of coronaviruses and the number of cases. Coronaviruses like SARS-COV-2 mutate continuously (this is why companies like deCODE can tell the geographical origin of different strains) as the molecular machinery for replicating their RNA genome is not very accurate. When you combine this with the millions of mild cases out in the world, the odds are on our side that there is at least one person infected with a strain that has a mutation that makes the virus less dangerous (attenuated). We just need to look for this strain – luckily the tools we need to use (genome sequencing) are now cheap and quick. What would have been impossible 20 years ago can now be done in a week.

Aren’t most people who have mild/asymptomatic cases infected with a dangerous strain?

Yes. Almost all (>99.9%) of people who are infected (and have a mild case) are infected with a dangerous strain of the virus, they just happen to have an immune system that can control the virus well. With COVID-19 a mild case does not mean you are infected with an attenuated strain – for most people with a mild case if they happen to infect a person with preexisting conditions or who is old, that person will be at a high risk of dying. A mild case does not equal a harmless strain.

My argument is coming from the other direction. While almost all mild cases of COVID-19 are caused by a dangerous strain of SARS-CoV-2, an attenuated strain of SARS-CoV-2 will only cause mild disease. If you want to find an attenuated strain you need to look at mild cases even though >99.9% of the people you check will be infected with a dangerous strain. What we want to find is one of the rare natural viral mutants that is attenuated. Where you will find such a viral mutant is in people with a mild/asymptomatic form of the disease.

Can’t we just wait for the vaccine?

No. Apart from the time it will take to develop, trial, and mass produce a vaccine (12-18 months), it is unlikely that any vaccine will be practicable. The reason why is immunity to respiratory viruses (like corona) doesn’t last long – 6 months to 2 years. We would have to keep vaccinating everyone in the world every year (or maybe every 6 months if we are unlucky). This just isn’t going to work in the real world (especially poor countries) and is one of the reasons we don’t have a vaccine for the coronavirus strains that cause the common cold. Unless we can drive the current dangerous SARS-CoV-2 strains to extinction we are going to have a problem with this disease indefinitely

Isn’t social distancing and quarantining solving the problem?

Yes and no. Yes countries like South Korea and Australia have shown that through mass screening and social distancing you can keep a lid on the disease, but this leaves the population susceptible to a new outbreak. Singapore and Japan have recently seen this in action where they eased restrictions and found the disease came back and they had to reintroduce restrictions. I don’t think many people want to live for years with cycles of restrictions, easings and further outbreaks.

Wouldn’t the use of such an attenuated strain just be a vaccine?

Yes in one way, but it is a little more subtle. Assuming we can find an attenuated strain, then how to best use it a separate question. The most important thing to note is that such a virus would not be a vaccine from a regulatory perspective. It is just a natural virus that you can catch in a natural way. Hang out someone infected with the attenuated virus and you will catch it without doing anything, go home and those around you will catch it from you and so on. While I wouldn’t suggest this is the best way to get the attenuated virus out into the community (it would be much better to just post a sample of the virus to everyone), such spread is outside the regulatory framework for vaccines.

Giving it deliberately to lots of people would change the ecosystem for the dangerous strains of the virus. The dangerous strains would find it difficult to spread through the community as many (most) people would have already been infected (and hence immune) with the attenuated strain. Overtime the dangerous strains would become rarer, and the attenuated strains more common, until eventually the dangerous strains would go extinct and we would just be left with the mild version. While we would not be able to get rid of this mild strain, it would just be another of the hundreds of viruses out there causing common colds. The idea is at its base is really to replace the dangerous strains with a less dangerous strain that we can live with.

Why can’t we just use the less pathogenic SARS-CoV-2 strain already identified in China?

While the ZJ01 SARS-CoV-2 strain identified in China appears to be less pathogenic, the mutations that make it so are single base changes. These can easily mutate back to the more dangerous version of the virus. The viral strain we want to find will have a deletion mutation where a section of the viral genome is removed. Deletion mutations are much more difficult to mutate back to the dangerous type since rather than just a change from one nucleotide to another (e.g. C > T), the deleted region is missing and can’t easily be recreated by mutation. Put simply, deletion mutations are more stable to back reversion.

Which Came First – The Chicken or the Egg?

One of the great things about having children is it reacquaints you with things you have not thought about for a long time. The old Chicken or the Egg paradox is one of those classic brain teasers that children of a certain age love. It is a really good one since the answer depends on how you parse the question. I thought I would list all the different answers my children and I could come up with.

Evolution 1 – Chicken

The first Chicken had to have hatched from an egg laid by a proto-chicken (i.e. a bird that was very similar to a chicken, but not actually a chicken). This means that the Chicken came before the first chicken Egg since only a chicken can lay a chicken egg.

Evolution 2 – Egg

If we consider that a chicken egg is an Egg that a Chicken hatches from then the Egg must comes first. It might have been laid by a Proto-Chicken, but out of this Egg hatched a Chicken.

Evolution 3 – Egg

The Egg is a much older than Chickens. What we now recognise as Eggs first appeared at least 300 million years ago. This was long before the first Chicken which is a domesticated version of the Indian Red Jungle Fowl from sometime in the last 10,000 years.

Evolution 4 – Unanswerable

Given that the definition of what separates a Chicken from a Proto-Chicken is undefined, it is not possible even in theory to say when the first Chicken hatched even if we had access to a time machine. If we can’t know when the first Chicken hatched we can’t answer the question.

Biblical – Chicken

According to Genesis 1 God created all the animals on Day 5 therefore the Chicken was created before the first Egg. It is an open question if the first Chickens were created with full formed eggs inside them and so if the first Egg was laid on Day 5 or not.

Word Order – Chicken

In the question “Which came first the Chicken or the Egg?”, the word Chicken precedes the word Egg.

Word Origin – Egg

The word Egg comes from Old Norse and ultimately back to the Proto-Germanic and before that Proto-Indo-European. It is a much older word than Chicken which is an Old English word of unknown origin.

English Language – Chicken

The original word for Egg in Old English was Ey and only in the development of Middle English did the Norse word egg become the common term. The word Chicken is from Old English and so it appeared first in the English language.

Dictionary – Chicken

In the English Dictionary the letter C comes before the letter E hence Chicken is first. The same applies to Encyclopaedias, although of course no child of today knows what an Encyclopaedia is.

Wikipedia – Chicken

The first entry for Egg was in 2005 while the first entry for Chicken was in 2004. Who would have guessed?

Finish Line – Chicken

In a race a Chicken will always beat an Egg to the finish line.

Drop Test – Egg

Chickens can fly so if you drop an Chicken and Egg off a barn roof together the Egg will hit the ground first. Chickens are surprising good flyers once they are allowed out to roam around for a few months.

There must be more !

The Last Word on Free Will

People have been arguing if free will exists or not for millennia with little progress. You have the Incompatibilists on one side arguing that free will and a deterministic universe can’t both be true, and the Compatibilists arguing that they can. While the heavy artillery appears to be on the side of Incompatibilism (the universe does appear to be deterministic), the inherent nihilism of Incompatibilism has meant most people have opted for some flavour of Compatibilism of varying sophistication. The arguments for both sides washes back and forth and we are no closer to an answer than the ancient Greeks.

Rather than approaching the question of free will from a philosophical perspective, we can just approach it empirically.

  1. The probability free will exists is greater than zero. Our knowledge of the universe is incomplete and so no matter how much evidence there is supporting a belief we can not apply a probability of zero to any hypothesis that negates this belief. All the evidence suggest fairies don’t exist at the bottom of the garden, but there is some finite probability that they do. In the case of free will this means that while all the evidence points to it not existing, we can not say with certainty it does not exist.
  2. If there is no free will then it meaningless what beliefs you hold about free will. There is nothing lost in life believing in free will if it doesn’t exist since whatever beliefs you have they were predetermined.
  3. If there is free will then believing there is no free will is throwing away your life. If free will exists and you go through life believing everything is predetermined then you will have missed making the choices open to you by free will. You may spend your life in a nihilistic funk when you could have chosen differently.

Given these three statements the only conclusion we can reach is we have to live as though free will exists even if everything we know points to it not existing. Nothing is lost believing in free will if it doesn’t exist, while everything of importance is lost if you don’t believe in free will and it exists. No matter how unlikely free will is, and it appears very unlikely, the conclusion doesn’t change – as long as our knowledge of the universe is incomplete the only rational action is to live as though free will exists.

While I am sure that I am not the first person to propose this solution to the free will problem, I have not been able to find who first proposed it. If anyone knows the source of this argument please leave a comment.

Easy Creme Brulee

Easy Creme Brulee

Easy Creme Brulee

For something so simple as creme brulee I wondered why so many recipes turn four ingredients and few basic steps into a cooking challenge. Here is my easy recipe for creme brulee adapted for Australian conditions. The key to success is to use high quality ingredients and avoid curdling the custard by overcooking. Vanilla bean paste tastes and looks the same as using whole vanilla beans, but it is a whole lot easier. Follow these simple steps and it is hard to go wrong.

Ingredients

  • 600 mL (1 large carton) of fresh single cream (35% milk fat).
  • 6 quality eggs.
  • 1/3 cup (75 g) of white sugar.
  • 1 teaspoon of vanilla bean paste (important).

Method

  • Preheat the oven to 160˚C. Boil 2 L of water in an electric kettle.
  • Pour cream into a small saucepan and add the sugar. Heat slowly to the point just before boiling (don’t let the cream boil).
  • While the cream is heating seperate 6 egg yolks into a medium size bowl and add the vanilla bean paste. Mix with a hand whisk (or fork) until the vanilla and yolks are combined (10 seconds).
  • Once the cream reaches near boiling, remove and pour slowly into the yolk mixture whisking gently the whole time. This should take about 20 seconds.
  • Pour the thin custard through a fine sieve into a glass 1 L or larger jug. This will remove any egg lumps and make it easer to pour into the ramekins (i.e. the small ceramic bowls).
  • Place 4 to 6 ramekins in a metal baking tray that is 10-15cm deep. Pour the custard into each of the ramekins and place the whole tray in the hot oven.
  • Pour in the boiling water until it comes 2 cm from the top of the ramekins. It is easier to pour in the boiling water when the tray is in the oven than trying to move a tray filled to the brim with boiling water.
  • Bake for 30 min. Remove the ramekins and let cool on the bench (30 minutes).
  • Wrap each ramekin with plastic wrap and place in the fridge for at least 3 hours.
  • Just before serving remove from the fridge, add two teaspoons of any sugar on top of the custard, shake gently to evenly distribute the sugar, and then blacken the sugar with a butane blowtorch. You can also use the oven grill, but it will tends to heat the custard a bit more than using a blowtorch – also blowtorches are fun.
  • Serve immediately to your impressed guests or family.

 

Bitcoin is being set up to fail spectacularly

Bitcoin Price

Bitcoin Price 2017

 

Bitcoin is all the rage at the end of 2017. The interesting question is not why it has risen so high and so fast, but why it has not been made illegal. The most impressive observation about Bitcoin (and its block-chain brethren) is it has been allowed to run free, sucking in all and sundry, most who have no idea what a block-chain is, but who know their friends and neighbours have made a fortune from it. The mass media has been neutral-to-supportive of the speculation.

The question is why those that control the monetary system (i.e. the rich and powerful) have allowed this run given the revolutionary nature of Bitcoin? If Bitcoin succeeds they will lose control of their wealth and power to a bunch of computer anarchists with a cool idea.

The powerful if they wanted could shutdown Bitcoin and all the other block-chain currencies tomorrow – when you are using as much electricity as a medium-sized country you can’t really hide. Despite the risk nothing has happened. They are not stupid (well the people advising them anyway), so it does not make much sense that Bitcoin has been allowed to continue.

The only rational hypothesis I have been able to come up with is the intention is to ensure the general population do not just feel indifferent to Bitcoin (this would be the result of an earlier crackdown), but that they must totally hate it and the whole concept of the block-chain. Hate Bitcoin so much that no future idea like this can ever gain popular support.

With this in mind, the rise of Bitcoin makes much more sense. When the inevitable crash comes it will burn a huge number of ordinary people who have been sucked into the hype and speculative mania. The aim appears to ensure that Bitcoin (and by association all block-chain currencies) are seen as the greatest scam of the last 100 years.

If what I am suggesting is true then Bitcoin has some way to run yet (my guess is at least six months). The risk of overturning the current monetary system (and the wealth and power than comes from controlling it) is far too great to ever let any alternative arise. Bitcoin has to more than fail – it has to fail spectacularly. Everything is on track to ensure this outcome and the pain from the economic fallout will be long and deep (for the little people anyway).

Damnatio memoriae

The last few days have reminded me that we really need to stop “glorifying” the actions of mass murders and actually do something to prevent others repeating their actions. Making losers famous by mentioning them in the mass media just encourages more losers. We can learn from the past and the Roman damnatio memoriae is an approach we would be wise to revive.

Rather than giving those that have stepped outside society’s boundaries fame, let us instead remove them from history. Total and utter obliteration. We have the technology and legal authority to completely delete the historical existence of someone evil. Remove everything about them; their birth, schooling, job history, marriage(s), relationships, photos, phone records, emails, Facebook posts, even the banal like credit records or receipts from Walmart. Remove everything about them such that they effectively never existed as a person. Leave nothing. If we need to refer to their actions then give them a pseudonym such as the “butcher of X”. In a 100 years there will be no record or memory they ever existed while we still remember their victims.

It might seem impractical to follow such a path given the ubiquity of modern media, but in practice it is easier today to remove someone from history than it has ever been. A single authority with determination can track down and remove every fragment of an individual’s existence.

We must do something rather than wring our hands in despair and let history repeat. Let today be the last time evil has a name.

Dead simple ssh login monitoring with Monit and Pushover

Following on from my earlier post on how to set up Dead simple CentOS server monitoring with Monit and Pushover, I recently added monitoring for ssh logins. I wanted to be able to see who is logging into my servers and be notified if anyone not authorised gained access. If you already have set up a Monit and Pushover system then this just requires adding of an extra monit.conf file.

Create the ssh logins monit .conf file with the following.

# nano /etc/monit.d/ssh_logins.conf

check file ssh_logins with path /var/log/secure 
  #Ignore login's from whitelist ip addresses
  ignore match "/var/www/ignore_ips.txt"
  if match "Accepted publickey" then exec "/usr/local/bin/pushover.sh"
  if match "Accepted password" then exec "/usr/local/bin/pushover.sh"

If you want to be able to ignore logins from certain IP addresses (i.e. your own) then create a text file with the list of IP address to be ignored (one per line).

# nano /var/www/ignore_ips.txt

123.123.134.123
122.121.121.121
...

Check that all the .conf file are correct

# monit -t

If everything is fine then restart monitoring by reloading the new .conf files.

# monit reload

Now anytime someone logs in to the server you will be sent notification. The only downside is that notification takes around a minute to occur since the notification is only pushed once monit checks the secure logfile. It is possible to get instant notification by using pam_exec, but that is another post.

Easy Protection of File Downloads in WordPress

027_1

I recently wanted to protect some files from unauthorised download on a WordPress site, but still allow authorised users to easily access to the files.

The simplest solution I found was to put the files in custom directory, place the links to the files on a WordPress password protected page, and use a .htaccess file to limit access to the files to users who are logged in. This rather simple approach works rather well if you take a little care with the directory and/or file naming.

Here is the step-by-step guide.

1. Make a new directory on your site and upload the files you want to protect to this directory (using ftp or scp). Make sure you chose a directory name that is hard to guess. I would recommend a random string — something like “vg4thbspthdbd8th” — just don’t use this exact string!

mkdir /path_to_protected_directory/

2. ssh into the server and and create a .htaccess file in the protected directory using nano.

sudo nano /path_to_protected_directory/.htaccess

3. Copy and paste the following text into the .htaccess file.

Options -Indexes
php_flag engine off
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?yourwebsite.com [NC]
RewriteCond %{HTTP_COOKIE} !^.*wp-postpass.*$ [NC]
RewriteRule \.(zip|rar|exe|gz)$ - [NC,F,L]

4. Change the yourwebsite.com to your website’s actual name. You should also change the RewriteRule line to suit the content you wish to protect. Just add the extensions of any file type you want to protect from unauthorised download.

That is it.

The major limitation with this approach is the download protection depends on the content of the user’s cookies. Since these can be faked by the technically knowledgeable, the protection is not perfect.

This is not as big a problem as it might first appear, because as long as you give the files and/or the directory non-obvious names, any unauthorised user will not know the required path to the files. They will only know the correct paths if they can log in, and if they can do this, they don’t need to fake any cookies.

While not perfect, this approach should work well for the casual protection of unauthorised downloads, but don’t use it for very sensitive files!

Carnot Efficient Dyson Spheres are Undetectable by Infrared Surveys

dyson_sphere

An interesting series papers were published in The Astrophysical Journal in 2014 by J. T. Wright and colleagues who used data from the WISE and Spitzer wide-field infrared astronomical survey data sets to try to detect Dyson spheres [1-3]. While very thought provoking, the entire premise of their study rested on the assumption that the Dyson spheres created by advanced civilisations will radiate waste heat around 290K [2:2.6.4]. This assumption allowed them to hypothesise that Dyson spheres radiating waste heat at this temperature would show up as very bright infrared sources well above the 15-50K background emission from interstellar gas and dust clouds [2:2.6.4].

Wright et al. provided no detailed reason for assuming this waste heat value other than the Carnot efficiency of a Dyson sphere around a sun-like star is 0.95 at 290K [2:2.6.3]. They felt that this was a “reasonable” value to use, since in their opinion, it balanced the materials required to build a Dyson sphere with the overall Carnot efficiency [2:2.6.4]. An important question that needs to be considered is would any advanced civilisation capable of constructing Dyson spheres throwaway 5% of the potential energy available if this waste could be avoided? If we assume they could build more efficient Dyson spheres, would it be possible for us to detect them in the infrared spectrum above the background noise?

The Carnot efficiency of a Dyson sphere is determined by the Carnot equation η = 1 − Tw / T where T is the temperature of the star (5800K for a star like our sun) and Tw is the temperature of the waste energy emitted by the sphere [2:2.6.3]. To achieve a 95% Carnot efficiency around sun-like star a Dyson sphere needs to have a radius approximately that of Earth’s orbit (i.e. 1 AU) [2:2.6.3].

As the spheres diameter grows larger, the waste energy temperature becomes lower and the efficiency higher. For example, to achieve a Carnot efficiency of 99%, the Tw would need to be ~58K assuming a sun-like star. For a Dyson sphere to radiate at this temperature it would need to have a surface area 625 times greater than one that radiates at 290K (see equation 12 of [2]). This efficiency corresponds to a sphere with a radius of ~25 AU around sun-like stars.

For reasons unknown, Wright et al. decided to use a Carnot efficiency of 99.5% (with a corresponding Tw of 29K) in their counter example as to why 95% was a reasonable efficiency for any Dyson sphere building civilisation to use. They calculated that the sphere surface area to achieve this Carnot efficiency would need to have a surface area 10,000 times larger (100AU radius), but assumed that a Dyson sphere of this size would be impractical and hence only spheres with an efficiency of 0.95 would be built.

This is an unusual assumption to make since it means any advanced civilisation capable of building a Dyson sphere would have to waste 5% of the potential energy available. A 0.99 or better Carnot efficient sphere could be built using only a small fraction of the material resources available within our solar system [2]. If you are civilisation able to build a Dyson sphere the size of Earth’s orbit, then you would be able to build one larger and much more efficient using a relatively small increase in resources and time.

The consequences of this 0.95 efficiency choice is not minor. If Wright et al. had assumed Dyson spheres are 0.99 (or better) Carnot efficient then their emission spectra would not be detectable above the background infrared emissions of interstellar gas and dust – put simply, the emission signal from efficient Dyson spheres will be swamped by infrared noise in any wide-field infrared surveys.

Unfortunately this means that all we can conclude from Wright et al. study is that there are few (or no) Dyson spheres built with a 0.95 (or less) Carnot efficiency. If Dyson spheres do exist, and they are efficient (which we should expect of any advanced civilisation capable of building such spheres), we won’t be able to spot them via infrared astronomical surveys. The good news there is a different approach for finding efficient Dyson spheres, but that is another post.

References

1. Wright, J. T., Mullan, B., Sigurdsson, S., & Povich, M. S. (2014). THE Gˆ INFRARED SEARCH FOR EXTRATERRESTRIAL CIVILIZATIONS WITH LARGE ENERGY SUPPLIES. I. BACKGROUND AND JUSTIFICATION. The Astrophysical Journal: 792:26.

2. Wright, J. T., Griffith, R. L.,  Sigurðsson, S., Povich, M. S., Mullan, B. (2014). THE Gˆ INFRARED SEARCH FOR EXTRATERRESTRIAL CIVILIZATIONS WITH LARGE ENERGY SUPPLIES. II. FRAMEWORK, STRATEGY, AND FIRST RESULT. The Astrophysical Journal: 792:27.

3. Griffith, R. L., Wright, J. T., Maldonado, J., Povich, M. S., Sigurdsson, S., Mullan, B. (2014). THE Ĝ INFRARED SEARCH FOR EXTRATERRESTRIAL CIVILIZATIONS WITH LARGE ENERGY SUPPLIES. III. THE REDDEST EXTENDED SOURCES IN WISEThe Astrophysical Journal: 792:28.

Dead simple CentOS server monitoring with Monit and Pushover

150532

My company Nucleics has an array of servers distributed around the world to support our PeakTrace Basecaller. For historical reasons these servers are a mix of CentOS 6/7 VPS and physical servers supplied by three different companies. While the Auto PeakTrace RP application is designed to be robust in the face of server downtime, I wanted a dead simple monitoring service that would fix 99% of the server problem automatically and only contact me if there was something really wrong. After looking around all the paid services I settled on using a combination of Monit and Pushover.

Monit is an open source watchdog utility that can monitor other Linux services and automatically restart them if they crash or stop working. The great thing about monit is that you can set it up to fix things on its own. For example, if the server can be fixed by simply restarting apache then I want the monitoring service to just do this and only send me a message if something major has happened. I also wanted a service that would ping my phone, but where I could easily control it (i.e turn on/off, set away times, etc).

Pushover looked ideal for doing this. For a one off cost of $5 you can use the Pushover API to send up to 7500 message a month to any phone. It has lots of other nice features like quiet times and group notification. It comes with a 7 day free trial so you have time to make sure everything is going to work with your system before paying.

The only issue with integrating monit and pushover is that by default monit is set to email alert notices. Most of our servers don’t have the ability to email (they are slimmed down and are only running the services needs to support PeakTrace). Luckly, monit can also execute scripts so I settled on the alternative approach of calling the Pushover API via an alert script that would pass through exactly what server and service was having problems. This alert script is set to only be called if monit cannot fix the problem by restarting the service. After a bit of experimentation I got the whole system running rather nicely.

Here is the step-by-step guide. I did all this logged in as root, but if you don’t like to live on the edge just put sudo in front of every command.

Setting up Pushover

After registering an account at Pushover, and downloading the appropriate app for your phone (iOS or android), you need to set up a new pushover application on the Pushover website.

Click on Register an Application/Create an API Token. This will open the Create New Application/Plugin page.

  • Give the application a name (I called it Monit), but you call it anything you like.
  • Choose “script” as the type.
  • Add a description (I called it Monit Server Monitoring).
  • Leave the url field blank.
  • If you want you can add an icon, but you don’t need to do this. It is nice though having an icon when you get a message.
  • Press the Create Application button.

You need to record the new application API Token/Key as well as your Pushover User Key (you can find this on the main pushover page if you are logged in). You will need both these keys to have monit be able to ping Pushover via the alert script.

Install Monit

Install the EPEL package repository.

# yum install -y epel-release

Install monit and curl.

# yum install -y monit curl

Set monit to start on boot and start monit.

# chkconfig monit on && service monit start

You can edit the monif.conf file in /etc but the default values are fine. Take a look at the monit man page for more details about what you might want to change.

Create the Pushover Alert Script

You need to create the script that monit will call when it raises an alert.

# nano /usr/local/bin/pushover.sh

Paste the following text substituting your own API Token and User Keys before saving.

#!/bin/bash
 /usr/bin/curl -s --form-string "token=API Token" \
 --form-string "user=User Key" \
 --form-string "message=[$MONIT_HOST] $MONIT_SERVICE - $MONIT_DESCRIPTION" \
 https://api.pushover.net/1/messages.jsonop

Make the script executable.

# chmod 700 /usr/local/bin/pushover.sh

Test that the script works. If there are no issues the script will return without error and you will get an short message in the Pushover phone app almost immediately.

# /usr/local/bin/pushover.sh

Configure Monit

Once you have the pushover.sh alert script set up you need to create all the service-specific monit  .conf files. You can mix and match these to suit the services you are running on your server. The aim is to have monit restart the service if there are any issues and only if this does not solve the problem, call the pullover.sh alert script. This way most servers will fix themselves and you only get contacted if something catastrophic has happened.

system

# nano /etc/monit.d/system.conf

check system $HOST
if loadavg (5min) > 4 then exec "/usr/local/bin/pushover.sh"
if loadavg (15min) > 2 then exec "/usr/local/bin/pushover.sh"
if memory usage > 80% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if swap usage > 20% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage (user) > 90% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage (system) > 80% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage (wait) > 80% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage > 200% for 4 cycles then exec "/usr/local/bin/pushover.sh"

apache

# nano /etc/monit.d/apache.conf

check process httpd with pidfile /var/run/httpd/httpd.pid
start program = "/etc/init.d/httpd start" with timeout 60 seconds
stop program = "/etc/init.d/httpd stop"
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then exec "/usr/local/bin/pushover.sh"
if failed port 80 for 2 cycles then restart
if 3 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

sshd

# nano /etc/monit.d/sshd.conf

check process sshd with pidfile /var/run/sshd.pid
start program "/etc/init.d/sshd start"
stop program "/etc/init.d/sshd stop"
if failed port 22 protocol ssh then restart
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

fail2ban

# nano /etc/monit.d/fail2ban.conf

check process fail2ban with pidfile /var/run/fail2ban/fail2ban.pid
start program "/etc/init.d/fail2ban start"
stop program "/etc/init.d/fail2ban stop"
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

syslog

# nano /etc/monit.d/syslog.conf

check process rsyslog with pidfile /var/run/syslogd.pid
start program "/etc/init.d/rsyslog start"
stop program "/etc/init.d/rsyslog stop"
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

crond

# nano /etc/monit.d/crond.conf

check process crond with pidfile /var/run/crond.pid
start program "/etc/init.d/crond start"
stop program "/etc/init.d/crond stop"
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

mysql

# nano /etc/monit.d/mysql.conf

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

Check that all the .conf file are correct

# monit -t

If everything is fine then start monitoring by loading the new .conf files.

# monit reload

Check the status of monit by using

# monit status

This should give you something like this depending on which services you are monitoring.

The Monit daemon 5.14 uptime: 3d 20h 17m

System 'rps.peaktraces.com'
 status Running
 monitoring status Monitored
 load average [0.00] [0.12] [0.11]
 cpu 0.2%us 0.1%sy 0.0%wa
 memory usage 106.6 MB [10.7%]
 swap usage 0 B [0.0%]
 data collected Tue, 19 Jul 2016 04:16:06

Process 'rsyslog'
 status Running
 monitoring status Monitored
 pid 1016
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 0
 memory 3.4 MB
 memory total 3.4 MB
 memory percent 0.3%
 memory percent total 0.3%
 cpu percent 0.0%
 cpu percent total 0.0%
 data collected Tue, 19 Jul 2016 04:16:06

Process 'sshd'
 status Running
 monitoring status Monitored
 pid 1176
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 4
 memory 1.2 MB
 memory total 20.7 MB
 memory percent 0.1%
 memory percent total 2.0%
 cpu percent 0.0%
 cpu percent total 0.0%
 port response time 0.006s to [localhost]:22 type TCP/IP protocol SSH
 data collected Tue, 19 Jul 2016 04:16:06

Process 'fail2ban'
 status Running
 monitoring status Monitored
 pid 1304
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 0
 memory 30.2 MB
 memory total 30.2 MB
 memory percent 3.0%
 memory percent total 3.0%
 cpu percent 0.1%
 cpu percent total 0.1%
 data collected Tue, 19 Jul 2016 04:16:06

Process 'crond'
 status Running
 monitoring status Monitored
 pid 1291
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 0
 memory 1.2 MB
 memory total 1.2 MB
 memory percent 0.1%
 memory percent total 0.1%
 cpu percent 0.0%
 cpu percent total 0.0%
 data collected Tue, 19 Jul 2016 04:16:06

Process 'httpd'
 status Running
 monitoring status Monitored
 pid 20963
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4h 5m
 children 2
 memory 7.7 MB
 memory total 19.0 MB
 memory percent 0.7%
 memory percent total 1.9%
 cpu percent 0.0%
 cpu percent total 0.0%
 data collected Tue, 19 Jul 2016 04:16:06

Suggestions

You may want to adjust the system.conf values if your server is under sustained high loads so as to scale back on the pushover triggers. Since you will know exactly what is the trigger this is quite easy to do.

To create a monit .conf file for a new services you just need to make sure that you use the correct .pid file path for the service and that the start and stop paths are correct. These can be a little non-obvious (look at syslog.conf for example). If you do make a mistake monit -t and monit status will show you what is wrong.

Once you have all this in place then sit back, relax and let the servers take care of themselves (well we can all dream).

Edit July 2017. I have been using this system for over a year now and it has been working great. I have had no problem that monit has not fixed by itself by just restarting the service. About the only issue I have had is load spikes on the server caused by a runaway service not monitored.

I have recently used the same approach to monitor for unauthorised logins which I wrote up Dead simple ssh login monitoring with Monit and Pushover.