Easy Creme Brulee

Easy Creme Brulee

Easy Creme Brulee

For something so simple as creme brulee I wondered why so many recipes turn four ingredients and few steps into a cooking challenge. Here is my easy recipe adapted for Australian conditions. The key to success is to use high quality ingredients and avoid curdling the custard by overcooking. Follow these simple steps and it is hard to go wrong.

Ingredients

  • 600 mL (1 large carton) of fresh single cream (35% milk fat).
  • 6 quality eggs.
  • 1/3 cup (75 g) of white sugar.
  • 1 teaspoon of vanilla bean paste (important).

Method

  • Preheat the oven to 160˚C. Boil 2 L of water in an electric kettle.
  • Pour cream into a small saucepan and add the sugar. Heat slowly to the point just before boiling (don’t let the cream boil).
  • While the cream is heating seperate 6 egg yolks into a medium size bowl and add the vanilla bean paste. Mix with a hand whisk until the vanilla and yolks are combined (10 seconds).
  • Once the cream reaches near boiling, remove and pour slowly into the yolk mixture whisking gently the whole time. This should take about 20 seconds.
  • Pour the thin custard through a fine sieve into jug. This will remove any egg lumps and make it easer to pour into the ramekins (i.e. the small ceramic bowls).
  • Place 4 to 6 ramekins in a baking tray which is 10-15cm deep. Pour the custard into each of the ramekins and place the whole tray in the oven. Pour in the boiling water until it comes up the side 2cm from the top of the ramekins. It is easier to pour in the boiling water when the tray is in the oven than try to carefully transport a tray full to the brim with boiling water.
  • Bake for 30 min. Remove the ramekins and let cool on the bench (30 minutes).
  • Wrap each ramekin with plastic wrap and place in the fridge for at least 3 hours.
  • Just before serving remove from the fridge, add two teaspoons of any sugar on top of the custard, shake gently to evenly distribute the sugar, and then blacken the sugar with a butane blowtorch. You can also use the oven grill, but it will tends to heat the custard a bit more than using a blowtorch – also blowtorches are more fun.
  • Serve immediately to your impressed guests or family.

 

Bitcoin is being set up to fail spectacularly

Bitcoin Price

Bitcoin Price 2017

 

Bitcoin is all the rage at the end of 2017. The interesting question is not why it has risen so high and so fast, but why it has not been made illegal. The most impressive observation about Bitcoin (and its block-chain brethren) is it has been allowed to run free, sucking in all and sundry, most who have no idea what a block-chain is, but who know their friends and neighbours have made a fortune from it. The mass media has been neutral-to-supportive of the speculation.

The question is why those that control the monetary system (i.e. the rich and powerful) have allowed this run given the revolutionary nature of Bitcoin? If Bitcoin succeeds they will lose control of their wealth and power to a bunch of computer anarchists with a cool idea.

The powerful if they wanted could shutdown Bitcoin and all the other block-chain currencies tomorrow – when you are using as much electricity as a medium-sized country you can’t really hide. Despite the risk nothing has happened. They are not stupid (well the people advising them anyway), so it does not make much sense that Bitcoin has been allowed to continue.

The only rational hypothesis I have been able to come up with is the intention is to ensure the general population do not just feel indifferent to Bitcoin (this would be the result of an earlier crackdown), but that they must totally hate it and the whole concept of the block-chain. Hate Bitcoin so much that no future idea like this can ever gain popular support.

With this in mind, the rise of Bitcoin makes much more sense. When the inevitable crash comes it will burn a huge number of ordinary people who have been sucked into the hype and speculative mania. The aim appears to ensure that Bitcoin (and by association all block-chain currencies) are seen as the greatest scam of the last 100 years.

If what I am suggesting is true then Bitcoin has some way to run yet (my guess is at least six months). The risk of overturning the current monetary system (and the wealth and power than comes from controlling it) is far too great to ever let any alternative arise. Bitcoin has to more than fail – it has to fail spectacularly. Everything is on track to ensure this outcome and the pain from the economic fallout will be long and deep (for the little people anyway).

Damnatio memoriae

The last few days have reminded me that we really need to stop “glorifying” the actions of mass murders and actually do something to prevent others repeating their actions. Making losers famous by mentioning them in the mass media just encourages more losers. We can learn from the past and the Roman damnatio memoriae is an approach we would be wise to revive.

Rather than giving those that have stepped outside society’s boundaries fame, let us instead remove them from history. Total and utter obliteration. We have the technology and legal authority to completely delete the historical existence of someone evil. Remove everything about them; their birth, schooling, job history, marriage(s), relationships, photos, phone records, emails, Facebook posts, even the banal like credit records or receipts from Walmart. Remove everything about them such that they effectively never existed as a person. Leave nothing. If we need to refer to their actions then give them a pseudonym such as the “butcher of X”. In a 100 years there will be no record or memory they ever existed while we still remember their victims.

It might seem impractical to follow such a path given the ubiquity of modern media, but in practice it is easier today to remove someone from history than it has ever been. A single authority with determination can track down and remove every fragment of an individual’s existence.

We must do something rather than wring our hands in despair and let history repeat. Let today be the last time evil has a name.

Dead simple ssh login monitoring with Monit and Pushover

Following on from my earlier post on how to set up Dead simple CentOS server monitoring with Monit and Pushover, I recently added monitoring for ssh logins. I wanted to be able to see who is logging into my servers and be notified if anyone not authorised gained access. If you already have set up a Monit and Pushover system then this just requires adding of an extra monit.conf file.

Create the ssh logins monit .conf file with the following.

# nano /etc/monit.d/ssh_logins.conf

check file ssh_logins with path /var/log/secure 
  #Ignore login's from whitelist ip addresses
  ignore match "/var/www/ignore_ips.txt"
  if match "Accepted publickey" then exec "/usr/local/bin/pushover.sh"
  if match "Accepted password" then exec "/usr/local/bin/pushover.sh"

If you want to be able to ignore logins from certain IP addresses (i.e. your own) then create a text file with the list of IP address to be ignored (one per line).

# nano /var/www/ignore_ips.txt

123.123.134.123
122.121.121.121
...

Check that all the .conf file are correct

# monit -t

If everything is fine then restart monitoring by reloading the new .conf files.

# monit reload

Now anytime someone logs in to the server you will be sent notification. The only downside is that notification takes around a minute to occur since the notification is only pushed once monit checks the secure logfile. It is possible to get instant notification by using pam_exec, but that is another post.

Easy Protection of File Downloads in WordPress

027_1

I recently wanted to protect some files from unauthorised download on a WordPress site, but still allow authorised users to easily access to the files.

The simplest solution I found was to put the files in custom directory, place the links to the files on a WordPress password protected page, and use a .htaccess file to limit access to the files to users who are logged in. This rather simple approach works rather well if you take a little care with the directory and/or file naming.

Here is the step-by-step guide.

1. Make a new directory on your site and upload the files you want to protect to this directory (using ftp or scp). Make sure you chose a directory name that is hard to guess. I would recommend a random string — something like “vg4thbspthdbd8th” — just don’t use this exact string!

mkdir /path_to_protected_directory/

2. ssh into the server and and create a .htaccess file in the protected directory using nano.

sudo nano /path_to_protected_directory/.htaccess

3. Copy and paste the following text into the .htaccess file.

Options -Indexes
php_flag engine off
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?yourwebsite.com [NC]
RewriteCond %{HTTP_COOKIE} !^.*wp-postpass.*$ [NC]
RewriteRule \.(zip|rar|exe|gz)$ - [NC,F,L]

4. Change the yourwebsite.com to your website’s actual name. You should also change the RewriteRule line to suit the content you wish to protect. Just add the extensions of any file type you want to protect from unauthorised download.

That is it.

The major limitation with this approach is the download protection depends on the content of the user’s cookies. Since these can be faked by the technically knowledgeable, the protection is not perfect.

This is not as big a problem as it might first appear, because as long as you give the files and/or the directory non-obvious names, any unauthorised user will not know the required path to the files. They will only know the correct paths if they can log in, and if they can do this, they don’t need to fake any cookies.

While not perfect, this approach should work well for the casual protection of unauthorised downloads, but don’t use it for very sensitive files!

Carnot Efficient Dyson Spheres are Undetectable by Infrared Surveys

dyson_sphere

An interesting series papers were published in The Astrophysical Journal in 2014 by J. T. Wright and colleagues who used data from the WISE and Spitzer wide-field infrared astronomical survey data sets to try to detect Dyson spheres [1-3]. While very thought provoking, the entire premise of their study rested on the assumption that the Dyson spheres created by advanced civilisations will radiate waste heat around 290K [2:2.6.4]. This assumption allowed them to hypothesise that Dyson spheres radiating waste heat at this temperature would show up as very bright infrared sources well above the 15-50K background emission from interstellar gas and dust clouds [2:2.6.4].

Wright et al. provided no detailed reason for assuming this waste heat value other than the Carnot efficiency of a Dyson sphere around a sun-like star is 0.95 at 290K [2:2.6.3]. They felt that this was a “reasonable” value to use, since in their opinion, it balanced the materials required to build a Dyson sphere with the overall Carnot efficiency [2:2.6.4]. An important question that needs to be considered is would any advanced civilisation capable of constructing Dyson spheres throwaway 5% of the potential energy available if this waste could be avoided? If we assume they could build more efficient Dyson spheres, would it be possible for us to detect them in the infrared spectrum above the background noise?

The Carnot efficiency of a Dyson sphere is determined by the Carnot equation η = 1 − Tw / T where T is the temperature of the star (5800K for a star like our sun) and Tw is the temperature of the waste energy emitted by the sphere [2:2.6.3]. To achieve a 95% Carnot efficiency around sun-like star a Dyson sphere needs to have a radius approximately that of Earth’s orbit (i.e. 1 AU) [2:2.6.3].

As the spheres diameter grows larger, the waste energy temperature becomes lower and the efficiency higher. For example, to achieve a Carnot efficiency of 99%, the Tw would need to be ~58K assuming a sun-like star. For a Dyson sphere to radiate at this temperature it would need to have a surface area 625 times greater than one that radiates at 290K (see equation 12 of [2]). This efficiency corresponds to a sphere with a radius of ~25 AU around sun-like stars.

For reasons unknown, Wright et al. decided to use a Carnot efficiency of 99.5% (with a corresponding Tw of 29K) in their counter example as to why 95% was a reasonable efficiency for any Dyson sphere building civilisation to use. They calculated that the sphere surface area to achieve this Carnot efficiency would need to have a surface area 10,000 times larger (100AU radius), but assumed that a Dyson sphere of this size would be impractical and hence only spheres with an efficiency of 0.95 would be built.

This is an unusual assumption to make since it means any advanced civilisation capable of building a Dyson sphere would have to waste 5% of the potential energy available. A 0.99 or better Carnot efficient sphere could be built using only a small fraction of the material resources available within our solar system [2]. If you are civilisation able to build a Dyson sphere the size of Earth’s orbit, then you would be able to build one larger and much more efficient using a relatively small increase in resources and time.

The consequences of this 0.95 efficiency choice is not minor. If Wright et al. had assumed Dyson spheres are 0.99 (or better) Carnot efficient then their emission spectra would not be detectable above the background infrared emissions of interstellar gas and dust – put simply, the emission signal from efficient Dyson spheres will be swamped by infrared noise in any wide-field infrared surveys.

Unfortunately this means that all we can conclude from Wright et al. study is that there are few (or no) Dyson spheres built with a 0.95 (or less) Carnot efficiency. If Dyson spheres do exist, and they are efficient (which we should expect of any advanced civilisation capable of building such spheres), we won’t be able to spot them via infrared astronomical surveys. The good news there is a different approach for finding efficient Dyson spheres, but that is another post.

References

1. Wright, J. T., Mullan, B., Sigurdsson, S., & Povich, M. S. (2014). THE Gˆ INFRARED SEARCH FOR EXTRATERRESTRIAL CIVILIZATIONS WITH LARGE ENERGY SUPPLIES. I. BACKGROUND AND JUSTIFICATION. The Astrophysical Journal: 792:26.

2. Wright, J. T., Griffith, R. L.,  Sigurðsson, S., Povich, M. S., Mullan, B. (2014). THE Gˆ INFRARED SEARCH FOR EXTRATERRESTRIAL CIVILIZATIONS WITH LARGE ENERGY SUPPLIES. II. FRAMEWORK, STRATEGY, AND FIRST RESULT. The Astrophysical Journal: 792:27.

3. Griffith, R. L., Wright, J. T., Maldonado, J., Povich, M. S., Sigurdsson, S., Mullan, B. (2014). THE Ĝ INFRARED SEARCH FOR EXTRATERRESTRIAL CIVILIZATIONS WITH LARGE ENERGY SUPPLIES. III. THE REDDEST EXTENDED SOURCES IN WISEThe Astrophysical Journal: 792:28.

Dead simple CentOS server monitoring with Monit and Pushover

150532

My company Nucleics has an array of servers distributed around the world to support our PeakTrace Basecaller. For historical reasons these servers are a mix of CentOS 6/7 VPS and physical servers supplied by three different companies. While the Auto PeakTrace RP application is designed to be robust in the face of server downtime, I wanted a dead simple monitoring service that would fix 99% of the server problem automatically and only contact me if there was something really wrong. After looking around all the paid services I settled on using a combination of Monit and Pushover.

Monit is an open source watchdog utility that can monitor other Linux services and automatically restart them if they crash or stop working. The great thing about monit is that you can set it up to fix things on its own. For example, if the server can be fixed by simply restarting apache then I want the monitoring service to just do this and only send me a message if something major has happened. I also wanted a service that would ping my phone, but where I could easily control it (i.e turn on/off, set away times, etc).

Pushover looked ideal for doing this. For a one off cost of $5 you can use the Pushover API to send up to 7500 message a month to any phone. It has lots of other nice features like quiet times and group notification. It comes with a 7 day free trial so you have time to make sure everything is going to work with your system before paying.

The only issue with integrating monit and pushover is that by default monit is set to email alert notices. Most of our servers don’t have the ability to email (they are slimmed down and are only running the services needs to support PeakTrace). Luckly, monit can also execute scripts so I settled on the alternative approach of calling the Pushover API via an alert script that would pass through exactly what server and service was having problems. This alert script is set to only be called if monit cannot fix the problem by restarting the service. After a bit of experimentation I got the whole system running rather nicely.

Here is the step-by-step guide. I did all this logged in as root, but if you don’t like to live on the edge just put sudo in front of every command.

Setting up Pushover

After registering an account at Pushover, and downloading the appropriate app for your phone (iOS or android), you need to set up a new pushover application on the Pushover website.

Click on Register an Application/Create an API Token. This will open the Create New Application/Plugin page.

  • Give the application a name (I called it Monit), but you call it anything you like.
  • Choose “script” as the type.
  • Add a description (I called it Monit Server Monitoring).
  • Leave the url field blank.
  • If you want you can add an icon, but you don’t need to do this. It is nice though having an icon when you get a message.
  • Press the Create Application button.

You need to record the new application API Token/Key as well as your Pushover User Key (you can find this on the main pushover page if you are logged in). You will need both these keys to have monit be able to ping Pushover via the alert script.

Install Monit

Install the EPEL package repository.

# yum install -y epel-release

Install monit and curl.

# yum install -y monit curl

Set monit to start on boot and start monit.

# chkconfig monit on && service monit start

You can edit the monif.conf file in /etc but the default values are fine. Take a look at the monit man page for more details about what you might want to change.

Create the Pushover Alert Script

You need to create the script that monit will call when it raises an alert.

# nano /usr/local/bin/pushover.sh

Paste the following text substituting your own API Token and User Keys before saving.

#!/bin/bash
 /usr/bin/curl -s --form-string "token=API Token" \
 --form-string "user=User Key" \
 --form-string "message=[$MONIT_HOST] $MONIT_SERVICE - $MONIT_DESCRIPTION" \
 https://api.pushover.net/1/messages.jsonop

Make the script executable.

# chmod 700 /usr/local/bin/pushover.sh

Test that the script works. If there are no issues the script will return without error and you will get an short message in the Pushover phone app almost immediately.

# /usr/local/bin/pushover.sh

Configure Monit

Once you have the pushover.sh alert script set up you need to create all the service-specific monit  .conf files. You can mix and match these to suit the services you are running on your server. The aim is to have monit restart the service if there are any issues and only if this does not solve the problem, call the pullover.sh alert script. This way most servers will fix themselves and you only get contacted if something catastrophic has happened.

system

# nano /etc/monit.d/system.conf

check system $HOST
if loadavg (5min) > 4 then exec "/usr/local/bin/pushover.sh"
if loadavg (15min) > 2 then exec "/usr/local/bin/pushover.sh"
if memory usage > 80% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if swap usage > 20% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage (user) > 90% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage (system) > 80% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage (wait) > 80% for 4 cycles then exec "/usr/local/bin/pushover.sh"
if cpu usage > 200% for 4 cycles then exec "/usr/local/bin/pushover.sh"

apache

# nano /etc/monit.d/apache.conf

check process httpd with pidfile /var/run/httpd/httpd.pid
start program = "/etc/init.d/httpd start" with timeout 60 seconds
stop program = "/etc/init.d/httpd stop"
if children > 250 then restart
if loadavg(5min) greater than 10 for 8 cycles then exec "/usr/local/bin/pushover.sh"
if failed port 80 for 2 cycles then restart
if 3 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

sshd

# nano /etc/monit.d/sshd.conf

check process sshd with pidfile /var/run/sshd.pid
start program "/etc/init.d/sshd start"
stop program "/etc/init.d/sshd stop"
if failed port 22 protocol ssh then restart
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

fail2ban

# nano /etc/monit.d/fail2ban.conf

check process fail2ban with pidfile /var/run/fail2ban/fail2ban.pid
start program "/etc/init.d/fail2ban start"
stop program "/etc/init.d/fail2ban stop"
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

syslog

# nano /etc/monit.d/syslog.conf

check process rsyslog with pidfile /var/run/syslogd.pid
start program "/etc/init.d/rsyslog start"
stop program "/etc/init.d/rsyslog stop"
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

crond

# nano /etc/monit.d/crond.conf

check process crond with pidfile /var/run/crond.pid
start program "/etc/init.d/crond start"
stop program "/etc/init.d/crond stop"
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

mysql

# nano /etc/monit.d/mysql.conf

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then exec "/usr/local/bin/pushover.sh"

Check that all the .conf file are correct

# monit -t

If everything is fine then start monitoring by loading the new .conf files.

# monit reload

Check the status of monit by using

# monit status

This should give you something like this depending on which services you are monitoring.

The Monit daemon 5.14 uptime: 3d 20h 17m

System 'rps.peaktraces.com'
 status Running
 monitoring status Monitored
 load average [0.00] [0.12] [0.11]
 cpu 0.2%us 0.1%sy 0.0%wa
 memory usage 106.6 MB [10.7%]
 swap usage 0 B [0.0%]
 data collected Tue, 19 Jul 2016 04:16:06

Process 'rsyslog'
 status Running
 monitoring status Monitored
 pid 1016
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 0
 memory 3.4 MB
 memory total 3.4 MB
 memory percent 0.3%
 memory percent total 0.3%
 cpu percent 0.0%
 cpu percent total 0.0%
 data collected Tue, 19 Jul 2016 04:16:06

Process 'sshd'
 status Running
 monitoring status Monitored
 pid 1176
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 4
 memory 1.2 MB
 memory total 20.7 MB
 memory percent 0.1%
 memory percent total 2.0%
 cpu percent 0.0%
 cpu percent total 0.0%
 port response time 0.006s to [localhost]:22 type TCP/IP protocol SSH
 data collected Tue, 19 Jul 2016 04:16:06

Process 'fail2ban'
 status Running
 monitoring status Monitored
 pid 1304
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 0
 memory 30.2 MB
 memory total 30.2 MB
 memory percent 3.0%
 memory percent total 3.0%
 cpu percent 0.1%
 cpu percent total 0.1%
 data collected Tue, 19 Jul 2016 04:16:06

Process 'crond'
 status Running
 monitoring status Monitored
 pid 1291
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4d 23h 33m
 children 0
 memory 1.2 MB
 memory total 1.2 MB
 memory percent 0.1%
 memory percent total 0.1%
 cpu percent 0.0%
 cpu percent total 0.0%
 data collected Tue, 19 Jul 2016 04:16:06

Process 'httpd'
 status Running
 monitoring status Monitored
 pid 20963
 parent pid 1
 uid 0
 effective uid 0
 gid 0
 uptime 4h 5m
 children 2
 memory 7.7 MB
 memory total 19.0 MB
 memory percent 0.7%
 memory percent total 1.9%
 cpu percent 0.0%
 cpu percent total 0.0%
 data collected Tue, 19 Jul 2016 04:16:06

Suggestions

You may want to adjust the system.conf values if your server is under sustained high loads so as to scale back on the pushover triggers. Since you will know exactly what is the trigger this is quite easy to do.

To create a monit .conf file for a new services you just need to make sure that you use the correct .pid file path for the service and that the start and stop paths are correct. These can be a little non-obvious (look at syslog.conf for example). If you do make a mistake monit -t and monit status will show you what is wrong.

Once you have all this in place then sit back, relax and let the servers take care of themselves (well we can all dream).

Edit July 2017. I have been using this system for over a year now and it has been working great. I have had no problem that monit has not fixed by itself by just restarting the service. About the only issue I have had is load spikes on the server caused by a runaway service not monitored.

I have recently used the same approach to monitor for unauthorised logins which I wrote up Dead simple ssh login monitoring with Monit and Pushover.

A Quick & Dirty Analysis of Apply HN

H4160-L83370839Y Combinator recently launched a new initiative where they asked the Hacker News community to identify promising startups to fund via the YC fellowship program.  This program provides US$20,000 to very early/idea stage startups to build their prototype/MVP. Potential participants were asked to post their concept to Apply HN and members of the HN community were asked to discuss the concepts and make ‘nice’ suggestions. The two best public applications will be funded by YC at the end of the month.

While a very interesting experiment in itself  (I really do applaud YC for actually trying new ideas in the VC world), the most fascinating aspect of this experiment is it give us an all too rare access to the VC pitch firehose. The investment community rarely (never) shares the raw data of what pitches they see come across their desk and we are left only seeing the end product (the startups they fund). As a founder of a startup you really have no way of knowing what competition you are facing for investor time and dollars. Is your startup the next great opportunity or just another lost cause doomed to failure?

To answer this question I did a quick and dirty analysis of all 194 applications (minus my own) as of 11.30am 12th April AEST. I read carefully through every application and all the comments (this took me a bit over 6 hours) and sorted the applications into one of ten categories (see below) on the basis of their investment potential (Figure 1). Many startups fell into more than one category (e.g. non profit and network required) and in these cases I sorted them on the basis of what I believed was the primary category. After completing the analysis I randomly selected 25 applications from the pool and blindly reclassified them. The two classifications agreed in every case giving me faith that while my classification process may be invalid, it is at least replicable.

Categorization of Apply HN applications by investment potential.

Figure 1. Categorization of Apply HN applications by investment potential.

Results

Non Profit (48)

These startups were either explicitly not for profit, or implicitly not for profit in the sense that there was no way of them ever making a profit from their product or service. Many were attempts to scratch an itch of the founder(s), but none in this category appeared to answer the most fundamental question any investor will ask – how are you going to make money?

Network Required (41)

This was the second largest category. There were many startups proposed or begun that would be great business if the founders could get 10 million users — the problem is they all had no way to get to this point other than to hope if they build it people will come. I am highly sceptical that it is possible to build a mass market network based business today and most niches markets still exploitable are too small to offer the returns the investment community requires. It is a huge task to build a new network business – I am not saying it is impossible, but it is going to be very hard convincing investors you can do it and make a profit unless you have something really compelling.

Existing Players (31)

A surprising number of application were me-too startups with one or more strong existing competitors dominate the market and where the proposed offering was not at least 10 times better (the comments were great for drawing this out). It is fantastic to have a product where the market already exists since you don’t need to create a market, but your product needs to be significantly better if you want customers to switch. Just being a little better than the competition is not enough.

Lifestyle Business (30)

These I classified as having the potential to be good businesses, but not ever make the sort of returns required by the VC investor community. There were quite a few great startups and ideas in this category, but the accessible market (even allowing for later expansion) is just too niche. I am personally highly supportive of founders developing lifestyle businesses, but if your startup can only ever make a profit of a few million dollars a year (if everything goes right) it won’t interest most investors.

Feature Not Business (17)

In this category there were lots of great ideas, but they just weren’t big enough to sustain a business. You don’t want to build your business providing something that can be easily replicated by one of the big players.

Not Scalable (14)

It is fine to do things that don’t scale when you are building your startup, but if your processes can never be automated and will always need highly skilled labor the business will not be able to expand into a billion dollars business. These types of startups can make great lifestyle businesses if the margins are high, but trying to develop a non-automatable technology business which only offers low margins is a slow and nasty way to go broke.

Too Big (5)

These were ideas (some great) that were just too big for the YC fellowship program. If your startup is going to need $100 million to create the prototype then you are going to be in for a hard slog finding investors who will back you. The way to approach these sort of ideas is the way Elon Musk build Space X – start small in other businesses and as you gain credibility and success investors will then be willing to back you in your big ideas. Dream big, but take small steps.

Troll (5)

Not all applications were serious proposals, although some were amusing.

Biotech (2)

There were a couple of biotech/medical device startups in the list. This industry is notorious for losing money and unless you are very knowledgeable one that should be avoided as an investor.

Unicorn Embryo (0)

These were startups that had the potential to be worth over a billion dollars if everything went right. They needed to be tackling a multibillion dollar market in a technologically innovative manner and have a plausible plan on how to grow and defend this market. This is the sort of startup investors want to back. Unfortunately there were none.

I should add for conflict of interest reasons my own application was not assessed. I will leave it to others to decide what category it should be in.

Conclusions

I was most surprised to see how little emphasis applicants placed on eventual profitably. Yes it is fine to make a loss when launching, but you have to have a credible plan for how you will make a profit at some point in the future. I was also surprised to see how many network requiring applications seemed to have no viable plan for how to grow their network beyond making something cool and hoping the masses will come. Network based businesses are so valuable because they are so hard to create.

All in all this experiment has been very valuable and I thank YC for running it even if they don’t find a unicorn embryo to invest in.

Update

Apply HN ended with a short list of 20 applications and interestingly all 20 were from the 194 I reviewed here (there were a total of 343 applications). While the HN voting selection process ended in controversial circumstances over Pinboard, I was very honoured to have made the shortlist with my idea only application – TruSert.

CentOS 6: Adjusting the remote desktop resolution without a monitor being connected

H4160-L83370825
If you remote desktop share into a CentOS 6 system without a monitor being connected (i.e. a headless server) the screen resolution defaults to a tiny 800×600. Unfortunately the upstream provider (Redhat) has declined to provide system-config-display for EL6 (see this bug report), making it impossible to set the screen resolution in CentOS6 to something more useable as you would if you had a monitor attached.

The workaround is to use xrandr in a script to change the screen size. Here are the steps involved.

1. Open a terminal window and type xrandr. This will list your various screen devices.

xrandr

You should get something like this

Screen 0: minimum 320 x 200, current 800 x 600, maximum 8192 x 8192
VGA-0 disconnected  (normal left inverted right x axis y axis)
HDMI-0 disconnected (normal left inverted right x axis y axis)
DP-0 disconnected (normal left inverted right x axis y axis)

2. Type cvt with the screen dimensions you want (e.g. 1920 x 1200).

cvt 1920 1200

This should output something like this. Copy everything after “Modline”.

# 1920x1200 59.88 Hz (CVT 2.30MA) hsync: 74.56 kHz; pclk: 193.25 MHz
Modeline "1920x1200_60.00"  193.25  1920 2056 2256 2592  1200 1203 1209 1245 -hsync +vsync

3. Create a new shell script to call xrandr. This script will use xrandr to create a new mode, add the new mode to the screen device, and finally sets the screen output to the device using this mode.

nano setscreen.sh

Paste the copied cvt output information to after the  xrandr newmode line. Adjust the xrandr addmode line to be the name of your device (e.g. VGA-0) and the name of the new mode (e.g. 1920x1200_60.00) . Modify the xrandr output line to be this new mode. Your script should look something like this.

#!/bin/bash
xrandr --newmode "1920x1200_60.00" 193.25 1920 2056 2256 2592 1200 1203 1209 1245 -hsync +vsync
xrandr --addmode VGA-0 1920x1200_60.00
xrandr --output VGA-0 --mode 1920x1200_60.00

Save the script (control-x).

4. Change the script’s ownership and permissions to root and make it executable.

sudo chown root setscreen.sh
sudo chgrp root setscreen.sh
sudo chmod 755 setscreen.sh

5. Run the script to see if it works correctly.

sudo ./setscreen.sh

This should change the remote screen resolution to your desired size (e.g. 1920×1200). It can take a second or two to adjust.

6. To avoid having to run this script manually every time you log into the server, copy it into the /etc/X11/xinit/xinitrc.d/ directory.

sudo cp setscreen.sh /etc/X11/xinit/xinitrc.d/

Now that was so easy Redhat.

A Good Idea Checklist

Sky

I have had quite a bit of discussion about my article Ideas are not Cheap over the last few months and especially around the checklist for judging the quality of startup ideas.

The checklist was my personal list that I use to judge my own ideas and it consisted of 25 points in no particular order with no details. I thought I should do a follow up explaining these points in detail so they would be more useful for others. This rather long article is the result.

Caveats

I am biased towards startup ideas that can be bootstrapped as I believe most founders should be bootstrapping. Not everyone shares my beliefs, so I encourage you to create your own checklist. The real value of having such a list is in helping you think critically about your own ideas, not in claiming you met 15 out of 23 from my list.

Some of the criteria can really only be determined retrospectively (e.g. Criteria 5, 10, 11 & 22). These criteria are still valuable to consider as they help you think deeply about your ideas. You may not be certain if an idea meets a criteria or not, but you will still benefit from careful investigation.

Try to avoid erring on the generous side. If you are unsure if your idea meets a particular criteria then score it as no. Your ideas are your children, but unlike your real children your love for your ideas should be conditional.

There is no hard rule about how many criteria your idea should meet to be good idea. Even so, a rough rule of thumb is if you have less than 10 then keep working on your ideas and if it has more than 16 contact me so we can talk.

This checklist is ordered from most important to least important.

Good Idea Checklist

1. Can you execute the idea within your limitations?

This is the most critical criteria on this list. If the answer is no then the idea is not a good idea. You can have an amazing idea for a multi-billion dollar business, but if you can’t actually execute on it then put it aside and work on something else. For example, I hear many ideas for new social media startups that will be fantastic businesses if the person can get to 10 million users – the problem is they have no realistic plan on how to get to this number of users. Just building a social network that is better than every other social network does not mean people will use it. Facebook is not worth more than $250 billion because it is the best social network ever created, it is worth this amount because it has more than 1.5 billion users that advertisers pay to reach.

An idea you can’t execute on is worthless (to you).

2. Do you have a competitive advantage?

What is it about your idea that gives you an edge? Just being an avid user of SnapChat and Instagram does not give you a competitive advantage making a new photo sharing app. The question you have to ask yourself is do you know something really important about the problem or industry that no one else does? Why should you be the one to succeed when there are thousands of people who are smarter, richer, harder working, and better looking (sorry) that have tried and failed? Why has nobody executed successfully on your idea? If you don’t have good answers to all these questions then you don’t have a competitive advantage and your chances of success are slim.

The good news is you can acquire an edge by studying the problem and industry in depth. If you have an idea for an industry you are unfamiliar with then get a job in that industry. For example, if you have an idea for improving the buying and selling of homes then get your real estate license and work in an agency before launching your business. This will help you learn why things that seem crazy to outsiders are done and it will help you avoid building a product that nobody will buy. Selling will be easier as you will be able to speak the jargon of the industry and won’t come across as some starry-eyed dreamer who has no clue what the industry pain points are.

3. Can you bootstrap the idea?

By this I mean can you build the business without ever needing outside investors, not can you start the business, glue together a brittle demo, and go out and look for investors.

For an idea to be bootstrapped into a viable startup you need revenue above your burn rate before you run out of capital. Unless your kindly uncle is extremely wealthy, this means you need paying customers – ideally you will have your first customers before you launch. I see very few ideas that meet this requirement. If it were easy to build a startup by bootstrapping then there would be no such things as angel investors or venture capitalists.

One thing often overlooked by people when thinking about bootstrapping is they need an idea that will generate enough cash-flow to allow the startup to grow. Rapidly growing businesses require lots of cash and if your sales do not provide this cash-flow then you have a problem.  An idea is only a true bootstrap candidate if it can generate sufficient cash-flow to allow a startup to grow.

I should note that one of the huge advantages of bootstrapable ideas is they help you avoid building products that nobody wants, or more importantly, nobody wants to pay for. If you must get real paying customers right from the start then you can’t spend years and millions of dollars building something that no one needs, or pivoting from one bad idea to another while burning someone else’s money. Bootstrapped startups have to achieve product-market fit quickly or they die. Embrace the focus bootstrapping brings to your advantage.

4. Do you have profitable customers?

Unless you are running a charity, you need profitable customers. It is relatively easy to give something of real value away for free, but in the absence of very patient and understanding investors, you are going to need to make money sooner rather than later. Building a great product, getting users, and exiting by being acquired by a Facebook or Google is not a viable plan. You need profitable customers, or at least a very clear plan to get profitable customers, as early as possible. Never forget that users are not a substitute for customers.

You need to be able to identify and sell to profitable customers. It is desirable if your early customers are as homogeneous in their needs as possible, but you don’t want build on an idea that will only ever target a very narrow market. Many ideas are too narrow  in appeal to be good businesses. It is fine to have as your initial target market Welsh women between the ages of 25 and 35 with a passion for skydiving and needlepoint, but you need to have a plan for expanding your market into other areas such that you have a viable business.

Due to crazy spending fuelled by the venture capital industry, in some markets it is not possible to acquire profitable customers. If the players in your niche are burning truck loads of VC cash buying users then you may not be able to find profitable customers even if your product is far better. For example, in a lot of markets a new food delivery or ride sharing service just won’t be able to charge the true price of providing their service as there are too many competitors burning investor’s cash trying to get scale. Stay out of these markets until the money bonfire burns out.

5. Is the idea hard to replicate?

There is little point building a product that is easy to replicate since a better resourced team can and will come along and steal the market. You want to build products that very few people in the world have the skill or knowledge to make (i.e. where you have a competitive edge – Criteria 2). The only exception are markets where the first mover has a very significant advantage (Criteria 10), but even here being the first mover is just a factor that makes your idea hard to replicate.

6. Does the idea offers high margins that won’t be eroded over time?

This is related to Criteria 5, but it is slightly more subtle and in the long run more important. Startups with high margins can afford to make a lot of mistakes and still succeed. If your margins are 80% then you can mess up almost everything and still succeed, while if your margins are 3% you need to get (almost) everything right. High margins also solve cash-flow problems since every sale is basically money in the bank.

You want a build a business that can sustain high margins. With high margins you can afford to hire fantastic people and provide them with an environment where they will flourish. With a low margin businesses you have to be laser focused on every cent being spent. On top of this it is just so much more fun to work in a high margin business than a low margin one. You want to be worrying about how big to make the Christmas bonus this year, not if you can shave 0.1c off each unit by outsourcing the backend maintenance to some three man shop in Laos.

It is critically important that if you are going to build a startup around high margins that you have a credible plan to avoid your margins being eroded. Ask yourself very critically what is going to stop your margins from being driven down over time? The last thing you want is to build a high margin business (with a corresponding high cost base) and then have to later try to shift to a low cost model because your margins have evaporated. Very few companies can survive this transitions.

7. Does your sales model work?

Ask yourself if your sale model will work at the product’s price point. A high touch sales model (i.e. one with sales staff who chase leads) just does not work for many products if the customer lifetime value is under $1,000. A really common mistake people make is they see a huge demand in the market, know how to make a product that will meet this demand, but only later find they can’t sell the product for the price required to support the required sales model.

The small to medium business (SMB) market is notorious for leading people into this valley of death. There is massive demand by SMBs for better technology products and services — the problem is the lifetime customer value of most SMBs just can’t support the cost of acquiring them.

If you are need to use a high touch sales model to sell and support your product then you need to focus on large customers, or you need to work out a way to be able to sell your product using a low touch model. In some markets it is just not possible to close the gap between the low lifetime value of the customer with the high cost of acquiring the customer. Stay out of these markets no matter how attractive they appear on the surface.

8. Is your idea 100x better than any current product or service?

Most potential customers will stick with the devil they know rather than take the risk on something new. Your idea needs to not just be 10x better, it needs to be 100x better to make people change. Never forget that it has to be 100x better for the person making the decision, not the business or shareholders who will ultimately benefit. Plenty of great ideas have failed that would have helped a company make huge profits because they made life more difficult for the decision maker who had to decide to buy the product. Few decision makers ever lose their jobs for saying no to the new so you need to offer them something fantastic to get a yes.

9. Is your idea sticky?

Does your product or service provide such value to the customer that they will wonder how they lived without it? Will they find it impossible to switch to a new product or service because your product has become totally enmeshed in their processes and habits. If the answer is yes then you have the foundation for a great business.

Facebook is the classic model of this stickiness. Lots of people have made better social networks than Facebook, but the lock Facebook has through the network effect is incredible. If you are going to disrupt Facebook you had better have a product that is at least 100x better (Criteria 8). For a startup it is not likely to be possible to create such a product – even Google failed with Google+ and they had almost infinite resources.

The good news is thinking about the importance of stickiness will help you develop strategies to make your product stickier. Always be thinking about what can you change to lock in your customers, or what can you add to your product make it harder for them to switch to another product.

Just don’t go overboard with this process or you will find you can’t get any customers in the first place. Customers are aware that you want to lock them in and will resist your efforts – make a product too obviously sticky and they will stay away.

10. Does your idea have real and significant first mover advantage?

It is always desirable to have first mover advantage (assuming you are first), the problem is that in most markets first mover advantage does not exist. History is littered with examples of late entrants killing the original pioneers (look at what Amazon did to Book Stacks Unlimited). It is very easy to overestimate how much value there is in being first to market especially if you don’t have access to unlimited resources. If you really do have significant first mover advantage this can be very valuable, but be aware that it is likely a mirage.

11. Is your idea viral?

This criteria is both the second most important and the most difficult to determine in advance. If you have a true viral product of general appeal (i.e. if every new user recruits two or more new users) then all your problems are solved. You don’t need to make money (VC’s will chase you with a checkbook) and eventually a Facebook or Google will buy you out for billions.

The problem is it impossible to know if your product will be viral before launching, or even afterwards most of the time. It is very easy to convince yourself that your idea will go viral (i.e. all your friends say it is a work of genius and it is going to be H-U-G-E), but in most cases it does not happen. It is really, really hard to make a viral product with wide appeal.

If you believe your product will be viral then you need to ask yourself why a stranger would take the time to convince their friends and colleagues to use your product. When someone recommends your product they are investing their ego in your idea so it needs to make the recommender look uber cool / attractive / a genius in the eyes of the person to whom your product is being recommended. Very few ideas can do this. You should always assume, unless you have very good evidence to the contrary, that your product will not go viral beyond a very small niche.

One thing to be very wary of is self-limiting virality. Just like real biological viruses can spread rapidly through a sub-population, yet fail to spread through the general population (HIV is a classic example), ideas can also show the same limited virality. Your great idea might only appeal to 10,000 people in the world. You will launch and see your product spread virally through this niche (small communities talk) only to hit a wall after 6 months once everyone interested in it has been reached. Unless these 10,000 people provide enormous customer lifetime value you will have just created a product that some people love, but which won’t make money. Ask yourself how many people will love your idea so much that they will invest their ego in telling other about it.

12. Is the market the right size?

The right size depends on the funding strategy you are pursuing. If you are looking to raise investor money then the market needs to be in the billions, while if you are bootstrapping you want to avoid these large markets and concentrate on niche markets. The ideal size for a bootstrapped startup is a market around $25 million that will generate profits of $5 million per year. This is too small to attract VC-backed competitors, but large enough to be worth the hassle. There is no point going through the agony and stress of creating a startup if you are only going to earn $100,000 a year in profit. Think big, but not too big.

13. Does your idea fit into an existing market?

It is really hard to create a new product and a new market at the same time. It is much better if you build a product that fits into existing market and is 100x better (Criteria 8) than to be first to market and be banking on first mover advantage (Criteria 10).

14. Will existing market players be indifferent to your idea?

This criteria touches on a series of warnings signs that you want to avoid. Does your product have legal (patent/copyright/IP) issues? Will you threaten the business model of powerful player(s) who are willing and able to crush you? Does your product build on another businesses product or service that can change at any moment locking you out? If you answer is yes to any of these questions you need to have a carefully thought out strategies for how to avoid these dangers.

15. Does your idea create a monopoly?

Monopolies can be highly profitable and very desirable to own. The difficulty is, outside of small niches, that they can be very hard to create and even hard to defend. Ask yourself how you will be able to stop others entering your market? What strategies do you have for stopping a better resource player from stealing your customers?

One of the many advantages of bootstrapping is you can choose markets small enough that defend a monopoly is possible. The VC backed startups just won’t enter a market of $25 million a year. Provided you don’t get too greedy (or lazy) you should be able to retain a monopoly position once established in these small markets.

16. Is your idea simple to explain, but hard to conceive?

This is one of the most misunderstood points from my original article. People assumed that since their idea was hard for them to conceive, and since they can explain the idea, then it must meet this criteria. This is not what I meant (this is my fault). I considered removing this criteria from this list because of this confusion, but it such a valuable feature if true that I decided to keep it but explain it better.

So what do I mean by an idea being simple to explain, but hard to conceive? In some ways it is like a reversing a one-way hash function. It is easy to hash a string, but very difficult (impossible) to reverse the process. An idea that is hard to conceive is one that very few people could have come up with, either because you need to be a genius (unlikely), or they have a very unusual background with just the right combinations of skills and interests. Almost all ideas that are hard to conceive are also hard to explain while almost all ideas that are easy to explain are easy to conceive.

It is a very rare idea that is both hard to conceive and easy to explain. Coming up with one is like reversing a hash function. The value is that it is unlikely that anyone has thought of your idea before, but you don’t face all the problems involved in explain it  – you get all the benefits of complexity with all the benefits of simplicity.

17. Can your idea be sold using a low touch sales model?

If you don’t need to hire shiploads of sales staff to sell your product or service then, all things being equal, your life will be easier. Not only do you avoid all the difficulties involved in managing and motivating sales people, but many more markets will be open to you as your cost of acquiring a new customer will be much lower.

The difficulty is knowing if you can use a low touch sales model before launching. It is very common for people to think that their product will be easier to sell than it really turns out to be. You need to question why you think you can sell using a low cost sales model. Try to get as much advice and feedback from experienced sales people before deciding that this model will work. If products of similar cost and complexity are not sold via a low touch model then it is very probable that you won’t be able to use this model either.

18. Is your idea easy for customers to trial and use?

This one is obvious. Of course the easier it is for a customer to trial your product the more people will, and assuming it offers real value, the more people will buy. The difficulty comes from managing the expectations of potential customers for a product with complexity. If you make a complex product too easy to trial then potential customers can undervalue the product, or misunderstand what it can do. Sometimes less is more while other times more is less.

19. Can your idea be scaled at low cost?

It is common advice for startups to do things that don’t scale. While there is great value in this advice, it is not useful when applied to evaluating ideas. When deciding on the quality of an idea you want to choose ideas that can be scaled at low cost with minimal human labour. Whatsapp is an example of the sort of idea that scales at low cost, Magic is an example of the sort of idea that doesn’t. You really want to build a Whatsapp and not a Magic.

20. Does your idea need low cost labour?

You want to avoid ideas that require low cost labour for two reasons. First, most startups that need low cost labour are low margin businesses with all the problems this entails (Criteria 6). Second, managing large numbers of people is never fun. Unless you want to build an empire, you really want to hire as few people as possible to get the job done. Follow the path of Whatsapp which had 55 employees supporting 900 million users when it was acquired by Facebook. Every person you add to your business will only make your life more complex. Better one really good person paid $200,000 a year than four average people paid $50,000 each.

21. Is your idea of interest to the media?

Your idea is unlikely to be of interest to the media, and if it is, it is likely it will be for the wrong reasons. If you are lucky enough to have an idea that will generate positive press coverage this is great, but don’t lose too much sleep if it doesn’t. In the end it is customers that matter, not if your mother gets to see you on the cover of some magazine in a black turtleneck.

22. Can your idea be sold internationally?

Almost all products can be sold internationally, the real question to ask is your idea suited to international sales. Ideas based on geography, legal frameworks, and specific cultural interests or tastes are less desirable than ideas that have universal appeal. The difficulty with assessing the universal appeal of an idea is that it can be very hard to make this judgement since the world is a big place and we all have a specific and limited cultural frame of reference. It is much more difficult than it seems to know how much international appeal your idea will have and often you will only find out after launching.

23. Will your idea make a dent in the world?

We all only have one life so spend it doing things you care about. If you want to build a startup then build one that will make the world a little better than you found it. It is very unlikely that on your deathbed you will regret not making an extra $100,000, but it is much more likely you will regret not doing the things you really wanted to do. Seize your chances and live a full life.

Most importantly don’t stop thinking.