Sony Xperia Z3 Review

The Sony Xperia Z3

The Sony Xperia Z3 is my first non-Nexus phone in 2 years.  The Nexus 6, while nice, is too big a phone for me, so I was faced with a choice between the Android flagships: Sony Xperia Z3, Samsung Galaxy S5, LG G3, or the Motorola Moto X (HTC didn't bring their A game this year).  I've wanted a rugged phone for years now, so I knew I had to get the Xperia or Galaxy S5.  I eventually decided on the Xperia.  It was a good choice.

Overall Experience
The Xperia Z3 is an outstanding phone in pretty much every aspect. Camera, battery, skin, etc.  The one drawback is the dropped call issue I had which I'll describe in more detail below.

Build
This phone is rated IP 68, which means that it is dust proof and can be kept under 4.9 ft of fresh water for up to 30 minutes.  Although I'm very careful with my phones, this was a huge selling point to me: I could drop it in water and have it survive.

Charging with the charging port cover open

This ruggedness comes with one drawback: the phone's charging port has a cover.  This is
inconvenient not because you have to cover the port before going swimming with your phone, but because you have to cover the port before putting doing anything with the phone.  The port cover is very fragile when it's popped out for charging, so leaving the port uncovered is not an option.  You have to leave the port covered 24/7 when not charging the phone and pop it open when you want to charge the phone.  It's a minor annoyance.  Hopefully, the next iteration of the phone will allow you to tuck the port cover into another part of the phone so that you can leave the charging port uncovered unless you plan to get the phone wet.

This is where there should be a place to stick the port cover.

There is also a cover for the microSD and nano SIM ports, but unless you swap memory or SIM cards often, you can just leave that closed.

The phone feels expensive and nice, sporting what appears to be a glass back, although the edges are more plasticky.  The power/lock button is strangely located in the center of the right side and the volume buttons are unfortunately right below the power button.  This means your fingers are not going to be used to the button placement no matter what, unless you had an Xperia Z2, I suppose.

There's a dedicated camera button. Nice!  There's a dock port or something on the left side.

Why did Sony put the power button there? :(

Overall, looks and feels nice, despite awkward button placement and annoying charging port cover.  I look forward to taking underwater photos.

Camera
The camera on the Xperia Z3 is excellent.  The pictures are crisp and clear, and low light photos aren't too bad, so already it's better than a typical Nexus phone.  The shutter for the first shot isn't super fast, although it's quicker on manual mode at maybe a half-second.  Repeated shots are super fast once you take the initial shot.

Photo taken with the Xperia Z3

Photo taken with the Xperia Z3

Where the camera really shines is the number of modes.  Auto is default, but by switching to manual, you can control the resolution and aspect ratio.  I didn't notice a difference between 20 MP and 8 MP, though.  They have some fun filters like using your front-facing and back-facing cameras simultaneously, timeshifting video, 4K video, and some augmented reality filters.

Display
It's 2014 and all smartphone displays look the same to me.  For example, the LG G3 has a QHD screen, giving it a much higher pixel density than this (and all other) phone.  But even reviewers (e.g. The Verge) couldn't discern any noticeable advantage from the LG G3's super high-resolution screen.  I'm also indifferent about the display technology.  This phone is IPS LCD, which means it gets super bright and has excellent color reproduction.  It also has great contrast, although that is usually the domain of IPS's competitor Super AMOLED.  However, the Galaxy S5, a SAMOLED phone, also has excellent color production in addition to brightness and contrast.  What I'm saying is: it doesn't matter what high-end phone you get, the display will look awesome.

The one drawback of the display is that the phone is too big.  Technically, it's not the display that's the problem, it's the phone, but big displays mean big phones.  With my previous smartphones, from the EVO 3D's 4.96" x 2.56" front face to the Nexus 5's 5.43" x 2.72" face, I always grew accustomed to the ever-increasing size.  After two months with this 5.75-inch by 2.83-inch behemoth, I wouldn't mind a smaller phone, even if the display had to shrink to get there.  It's only too big by a slight margin; it's not a major drawback.  Perhaps I should have gotten the Xperia Z3 Compact.

Battery
The battery is what sells the phone for me.  The 3100 mAh battery keeps this phone chugging along for days. Yes, days.  I imagine the phone would last about 3 days with light use.  I routinely use it heavily in a day only to find it has over 50% battery at night.  It conserves battery excellently when not in use, losing less than 10% overnight.  So you can use it heavily one day, forget to charge it, and still have it last you the rest of the second day with light-to-medium use.

The phone's Power Management setting is a little inaccurate. It will tell you that it has 5 days left of battery at 100%, but that's only with light-to-no use.  However, if you need a phone to last you 5 days (e.g. going camping), this phone can conceivably achieve that, especially when you use one or more of it's power-saving modes (STAMINA Mode, Ultra STAMINA Mode, and Low Battery Mode).  I have STAMINA Mode set to turn on when my phone is at 20% battery.  It rarely gets used.

Performance
No lag. But it doesn't have Android Lollipop yet, which is reportedly laggy.

Data
All carriers and almost all phones now have LTE.  With T-Mobile I often pull 20-30 megabits per second.

Storage
I finally have a phone with external storage again.  The Sony Xperia Z3 can use microSD to add up to 128 GB of storage.  However, I haven't taken advantage of this yet, and I find the 32 GB internal memory sufficient.  This is largely because of cloud services such as Google Music that reduce my need for on-the-phone memory.  Still, the microSD slot is nice to have.

Calls
The calling is a bit disappointing.  This phone has two excellent features, Wi-Fi Calling and HD Voice, but they don't do much to improve the calling experience.

HD Voice calls are supposed to sound like CD-quality audio as opposed to phones' current radio-quality sound.  In order to make HD Voice calls on T-Mobile, both users have to have HD Voice-enabled phones, and both be using T-Mobile.  Note: I've noticed the HD Voice icon on some calls to landlines and, in the future, HD Voice will work across different carriers.  However, they don't sound much better than regular calls.  Which is fine, I suppose, I just hoped for more. (Note: Everyone else seems to disagree with me here)

WiFi Calling might actually be a detriment.  The Xperia Z3 has four phone settings: WiFi only, WiFi preferred, cellular preferred and cellular only (WiFi calling disabled).  Ideally, WiFi calling makes everything more convenient because you can make calls where you have no phone signal.  It also frees up cell phone towers, so other mobile users in the area should experience increased call quality and decreased dropped calls.  I also figured audio might sound better over WiFi calls.  If I can stream an HD movie over WiFi through Netflix, it should be simple to get CD-quality audio over WiFi.  However, the opposite happens.  WiFi calls sound tinny and quiet.  Another issue is that WiFi calling messes up my text reception.  When WiFi was enabled and I was using a 3rd party texting app, I didn't receive some texts until days later.  I'm not sure if the problem was the WiFi calling or using the 3rd party app with WiFi calling.  When WiFi calling is off, 3rd party texting apps work fine.

I'd like to leave my phone on WiFi preferred.  Instead I leave my phone on cellular only mode, unless I'm in an area with no reception and I need to make a call.  That aspect of WiFi calling works well.

Another issue: When I first bought the phone I also got a fair amount of dropped calls.  My Nexus 5 on the same network (T-Mobile) almost never dropped calls.  I got the Xperia Z3 and started dropping calls about once a day.  It hasn't happened in a month, though.  It may have just been a software issue.

Hopefully, software updates will fix calling, WiFi calling, and HD Voice, and make it possible to make perfect-quality calls from anywhere.

Operating System/Skin
I've preferred stock Android since the Samsung Galaxy Nexus, but in the absence of a reasonably-sized 2014 Nexus phone, I was forced to choose a manufacturer-skinned Android phone.  This phone has changed my mind on Android skins.  I thought that manufacturer skins were pointless after Ice Cream Sandwich, which finally made Android beautiful.  But the Sony Xperia Z3 user interface is good.  In fact, it's better than stock Android Kit Kat.

Caveats: The Xperia launcher is nothing special.  I immediately replaced it with Nova Launcher.  The Xperia keyboard is probably worse than stock Android.  The texting app is meh.  It's disappointing that there is an icon for NFC hogging up space on the notification bar.  I have to keep NFC turned off to keep a clean notification bar.  Similarly, there is a headphone icon hogging up space in the notification bar whenever headphones are plugged in, simply to tell me that headphones are connected. (Sony thinks I can't tell that there's a long black cord plugged into my phone?)

The power button menu

However, the power management software is pretty excellent, as I pointed out in the Battery section.  The camera app is cool, as I describe in the Camera section.  I love that the battery icon can optionally show a percentage; I used to use an app that created a second battery notification to get this functionality.  This creates some room for the extra notification icons that Xperia uses for NFC and headphones.  Screenshots are easy: simply holding down the power button presents a menu that has a screenshot and a record screen (!) button.  Record screen allows you to take video capture of your screen interactions, and it's pretty amazing.  WiFi calling has its own Android setting instead of being a specific app, which is cool.

The notification pull-down allows you to add and remove custom settings like WiFi, auto-rotate and flashlight widget.  I especially love the flashlight widget, which wasn't added to the stock Android notification pull-down until Lollipop.

There are also a bunch of Sony and T-Mobile apps like Movie Creator and T-Mobile My Account.  Most of these are useless bloatware, but a few have cool features.  The Playstation app that gives you the ability to play PS4 games on your phone is probably the most coveted app.  However, I don't have a PS4, so I couldn't test it out.  As usual, these apps can't be uninstalled, but you can turn off the features--like malware scanning notifications--so that you don't even know they're there.

Conclusion
Initial calling problems notwithstanding, the Sony Xperia Z3 is one of the best Android phones to date, if not the best.  The camera, battery life, performance, storage, and skin are best in class.  Being waterproof is a killer feature.  I imagine only the Xperia Z3 Compact, Samsung Galaxy S5 and Samsung Galaxy Note 4 can match its power and versatility, although the LG G3, and Motorola Moto X (2014) are excellent phones as well.

Policy: Our Broken Software Patent System (The Short Version)

I get it. Ain't nobody got time to read a 4,500-word article on software patents.  But I think this is an important topic, so I wrote a shorter version of my three-part article for impatient readers.

But first I have to plug two excellent sources of information about our broken software patent system: the This American Life episodes " When Patents Attack!" and " When Patents Attack ...Part Two". I cannot recommend these episodes of This American Life highly enough.

The Software Patent System Is Broken

The American patent system is completely broken, at least when it comes to software patents. Billion-dollar software companies are suing each other over minor pieces of code, patent trolls run rampant, and consumers are losing out as the original purpose of software patents have been lost.

The Point of Software Patents
Patents are a form of intellectual property, like copyright. Unlike copyright, patents cover inventions instead of creative works and last for a shorter time.  The idea behind patents is great: If you spend years inventing something, perhaps building a better mousetrap, then someone shouldn't be able to see your mousetrap in a store, steal your idea and sell the same mousetrap while bypassing all the hard work you did refining the improved mousetrap.  With patents, you must reveal your invention publicly, but then you have the exclusive rights to that invention for 20 years.

However, there are limits. You can't patent just anything, someone would patent the wheel and sue all the car manufacturers.  Patents must be a patentable thing (a process, machine, "[article] of manufacture," and composition of matter), and they must be  new, useful and nonobvious.

Applying these principles to software is pretty new. Software patents only became legal in the US in the 70's and 80's. The courts are still trying to figure it out.

Existing Patents Are Invalid - Non-New Patents
One problem with the current situation is that people are patenting things they shouldn't be able to patent. People are patenting existing software inventions.

During the "When Patents Attack!" episode of This American Life (TAL), the journalists from talked to David Martin, the founder of M-CAM. M-CAM uses special software to search through all existing software patents to find patents that are essentially the same. Ideally, each patent should be unique. However, sometimes the US Patent and Trademark Office (USPTO) grants patents for things that have already been patented. During the TAL episode, Martin looked up how many matching patents there were for one specific software patent. There were 5,303 matching patents.  Martin said that happens 30% of the time.

The specific software patent that Martin was looking up with TAL was Patent 5771354. According to the owners, Intellectual Ventures, it covers upgrading software on your home computer over the Internet.  When TAL looked at the text of the patent it seemed like it did a lot more than that. My own opinion is that it appears to cover doing any sort of data backup over the Internet, using an ID and password.

Drawing from Patent 5771354 showing
how all computer networks work

There are duplicate patents abound.  Furthermore, I'd argue that even if an inventor receives the patent on something, it doesn't mean he or she invented it.  That person may have just been the first to file a patent.  Filing a patent is an expensive, time-consuming process.  According to the USPTO, it takes about 2 years for a patent to be processed.  The cost of filing a software patent is around $10,000 (UpCounsel, Richards Patent Law, IPWatchdog) when including legal fees, which might be a drop in the bucket for a major corporation but is expensive for a small startup.

Existing Patents Are Invalid - Obvious Patents

Software patents might be obvious extensions of existing patents.  As a software engineer and a person who is skilled in the field, I would describe Amazon's 1-Click patent as fairly obvious.  As one blog pointed out, it's a "fairly broad concept" that the European Patent Office denied a patent to because it was "obvious to a skilled person".

Another example: Richard Stallman, a software engineer who is famous for his work on GNU, Emacs, and other free software, decries the obviousness of Patent 5963916, applied for in October 1996 in his text, "The Anatomy of a Trivial Patent".  Patent 5963916 covers listening to a preview clip of music on the Internet.  After posting a snippet of the patent, Stallman writes, "That sure looks like a complex system, right? Surely it took a real clever guy to think of this? No, but it took cleverness to make it seem so complex."  Stallman analyzes each line in the first portion of the patent to point out how each aspect of the idea was already existing or, at least, very obvious at the time.  The overly complex language commonly used in patents obscures the obviousness of the ideas.

Patents Aren't Serving Their Purpose - Patent Trolls
According to the most popular definition, a patent troll (also called a patent assertion entity) is a company that obtains patents--usually through buying them--and, instead of making products based on those patents, waits for another company to violate their patents, and then forces that company to pay licensing fees to use that patent.

Let's take the example used in the intro of "When Patents Attack!".  Since 1999, Jeff Kelling has been working for FotoTime, a small company that hosts a photo sharing website. In May 2008, they received a letter from FotoMedia (not to be confused with FotoTime) that said FotoTime was in violation of 3 patents.  FotoTime was told to pay licensing fees or risk going to court.  It turned out that FotoMedia had sent the letter to 130 other companies including Yahoo (Flickr), Shutterfly, Photobucket, and other companies, big and small.

Whatever patents were being violated, FotoTime had evidently come up with the same idea themselves, without reading FotoMedia's patent or copying some FotoMedia product.  Another weird thing was that FotoMedia had no photo-sharing website, so it wasn't a competitor to FotoTime.  When Kelling called FotoMedia to ask them which patents FotoTime was violating, FotoMedia wouldn't tell them until they went to court.

Kelling learned that fighting the patent in court would cost an estimated $2 to $5 million, which was "more than [FotoTime] could handle".  FotoTime settled with FotoMedia.

This is what patent trolls do.  They purchase a patent that is somewhat vague or perhaps even invalid, and instead of making some technology that uses the patent, they threaten to sue companies they think are using the patent.  Some companies will pay them a fee rather than spend millions of dollars fighting the patent.  The patent trolls will then use that money to purchase more patents and sue more people, over and over.

This isn't a business.  This is, as venture capitalist Chris Sacca described such practices on This American Life, "a mafia-style shakedown."

And patent trolls are growing bolder.  According to a Presidential study, the number of lawsuits brought by patent trolls more than doubled from 2010 to 2012, and these lawsuits accounted for 62% of all patent lawsuits in America in 2012.  Victims of patent trolls paid $29 billion in 2011.  Now in 2014, these numbers are probably higher.

Patents Aren't Serving Their Purpose - The Damage from Patent Trolls
Patent trolls kill innovation in a few ways:

1. Software companies must spend money on legal fees instead of growing their technology.  It's impossible to know whether your company will be violating a patent or whether a patent troll will target your company, so you must be prepared.
2. Some startups may not be able to pay the legal fees from a patent lawsuit and may go under.
3. Patent trolling can even create a chilling effect, causing programmers good ideas to never pursue their startup for fear of being sued by a patent troll.

Software patents no longer promote progress, if they ever did.  It's impossible to know whether a company is actually violating a patent because the patent office is a confusing mess, containing vague and duplicate patents.  And so any software company with a good idea can be sued at any time, by a patent troll claiming that software company is violating its patents.

The system needs fixing.

The Software Patent System Will Always Be Broken

Changing such a huge system to be efficient and coherent would be hard.  The software patent system is at the intersection of law and software engineering, an area few people understand well.  As such, it can be difficult to solve its problems or even describe what the problems are.

The Government Won't Fix the System
Firstly, the United States Patent and Trademark Office is unlikely to fix the software patent system.  For one, they are overburdened with work already. The USPTO has too many patents to process and too little time to process them.  I suspect that this extra amount of work has lead patent examiners to approve lower-quality patents.  A recent study also suggests that the patent office has lowered standards to cope with the huge backlog of pending patent applications.

Congress or the President could fix the system, but patent reform is not a sexy issue and is unlikely to get the attention it deserves.  Furthermore, I doubt the President or Congress understand this thoroughly complex problem.

The Industry Won't Fix the System
I doubt that software companies will fix the system.  Large software companies already have purchased hundreds or thousands of patents and are immune to patent trolls.  Small software companies are feel the most pain from this broken system, but they don't have the lobbyists to effect change in Washington.

Patent lawyers, whether they work for the government as examiners and judges or for companies as patent attorneys, have even less incentive to change the patent system.  The broken patent system is their livelihood, and a lucrative one at that.

The System is Intrinsically Unfixable
Another reason I believe the software patent system will always be broken is because the idea of software patents is absurd.  Software is inherently unpatentable.

Firstly, software is a fast-paced industry.  The next big thing is often deprecated in a short amount of time.  Let's look at smartphones for an example.  3G data was a huge selling point for the iPhone 3G in July 2008.  Two years later, HTC released the EVO 4G, the first 4G smartphone in the US.  However, that phone used WiMAX networking for its 4G capability, and WiMAX was declared dead as of 2013 or 2014.  All major US networks now use LTE for 4G data.  In less than 6 years, the way smartphones sent data over mobile networks changed multiple times.  Yet software patents last 20 years.  Twenty years is several lifetimes in software years, way too long to give exclusive rights to a software invention.

A second issue is that patents aren't as necessary for software as they are for hardware.  I understand the need for protecting hardware.  Inventing new mouse traps requires not just time, but materials.  You need to create lots of physical prototypes until you get it right, using real materials, like wood and metal.

"Inventing" software requires none of that.  All you need is one basic computer to program on and you can create endless prototypes without worrying about materials or trial runs in harsh conditions, unless you're trying to program for some specific device.  Even then, there are emulators that allow you to program apps for smartphones and vending machines, etc. without actually purchasing the device the software will end up on.

Lastly, the complexity of software causes difficulty in deciding their patent eligibility.  In Patent Failure: How Judges, Bureaucrats, and Lawyers Put Innovators at Risk by Bessen and Meurer, they describe how software's complexity hinders its patentability: "It is possible, however, that features of software technology make it particularly susceptible to the patenting of obvious ideas, especially given the legal doctrines of non-obviousness developed by the Federal Circuit."

Software is different.  Software might even be unpatentable.  But even if it isn't, the software patent system will always be broken because no one can fix it. That's why we must abolish software patents.

We Must Abolish (or Reform) Software Patents

The system is very broken, and why there's no hope that it will ever be fixed.  There are only two options: a sweeping change in software patent legislation written by multiple parties working in concert with software industry leaders to fix an abomination that shouldn't exist; ...or just save time and money by killing the whole software patent system.

Ending the System
It wouldn't be that bad to get rid of software patents.  Europe does not allow software patents (the European Patent Office has allowed some patents involving software, but "programs for computers" have not been regarded as patentable inventions since 1973), and they have a bustling software industry.

There would be many benefits to dismantling the software patent system.  First, there would no longer be any software patent trolls.  Second, software engineers would probably produce more free and open source software (FOSS) since they would no longer have to worry about securing software patents that could be used in defense.  This would create the conditions for a new software startup boom.  Without the fear of patent lawsuits and given new FOSS tools, starting a small software company would be less risky.  Even larger software companies would benefit, since they wouldn't have to waste resources on expensive legal fights.  Society would get more cool tech stuff released more often.  This increased innovation would probably have a positive effect on the American economy.

Let's not forget that there are already intellectual property protections for software.  Copyright will be the primary protection for software.  There are also trade secrets.  And lastly, there's the profit motive and the notoriety of releasing first.  In this fast-paced industry, whoever comes out with a cool new concept first will profit from having a working product in stores while other companies try to work out the kinks in their labs.

Reforming the System
I'm calling for the end software patents in the US and elsewhere.  However, if we cannot end the software patent system, we should reform it at the very least.   My list of reforms:

1. Software patents must be specific.
   No more language about "the machine", "network web sites", "memory bank portions" and other archaic language unused since the days of vacuum tubes for the sole purpose of obfuscation.  Requiring clear and current language without extraneous implementation details will make patent application, processing, and litigation go a lot faster and easier.
2. Software patents must be used.
   If you are the owner of the software patent, you must be using the invention to actually create a product, or you must license the patent at a reasonable price.  Waiting for someone to violate your unused patent so you can extort them is unethical and it should be illegal.
3. Software patents should have a short lifespan.
   Twenty years is too long to hold a monopoly on a technology in the software industry.  Bezos and O'Reilly suggested 3-5 years, which is much more reasonable.
4. The US Patent and Trademark Office must remove their bad patents.
   The Office must go through their existing patent database and get rid of duplicate patents and other invalid patents.  The USPTO is mostly to blame for this fiasco and they should fix this.  Otherwise, how can inventors tell what inventions to avoid using and applying for?

By reforming software patents--or better yet, getting rid of them altogether--, we can fix the problems in the industry before any more damage is caused.  When we do, society will reap the benefits.

Policy: We Must Abolish (or Reform) the Software Patent System

(Our Broken Software Patent System, Part 3 of 3)

We must abolish the software patent system.

Yeah, you heard me.  Scrap it.  Get rid of the whole thing.

I've already explained in utter detail why the system is very, very broken, and why there's no hope that it will ever be fixed.  There are only two options: a sweeping change in software patent legislation written by multiple parties working in concert with software industry leaders to fix an abomination that shouldn't exist; ...or just save time and money by killing the whole software patent system.

Ending the System
It wouldn't be bad to get rid of software patents.  Europe does not allow software patents*, and they have a bustling software industry.

There would be many benefits to dismantling the software patent system.  First, there would no longer be any software patent trolls.  Companies that abuse the legal system to prey on small software businesses would cease to exist.  Secondly, software engineers would probably produce more free and open source software (FOSS) since they would no longer have to worry about securing software patents that could be used in defense.  This would create the conditions for a new software startup boom.  Without the fear of patent lawsuits from patent trolls or even large legit software companies, and given new FOSS tools, starting a small software company would be a less risky concept.  As for larger software companies, abolishing software patents would make these companies more agile, since they wouldn't have to waste resources on expensive legal fights.  Society would get more cool tech stuff released more often.  All of this increased innovation would probably have a positive effect on the American economy.

Let's not forget that there are already intellectual property protections for software.  Copyright will be the primary protection for software.  No one can steal a software engineer's or software company's actual code.  There are also trade secrets.   Just like the Coca-Cola Company keeps their Coke recipe a trade secret, a software company can keep its compression algorithm secret if it so chooses.  And lastly, there's the profit motive and the notoriety of releasing first.  In this fast-paced industry, whoever comes out with a cool new concept first will profit from having a working product in stores while other companies try to work out the kinks in their labs.

I join Richard Stallman and the Free Software Foundation in calling to end software patents in the US and elsewhere.  However, if we cannot end the software patent system, we should reform it at the very least.

Reforming the System
The list of people that want to just reform the software patent system is longer than the end-patents-list (Jeff Bezos of Amazon and Tim O'Reilly of O'Reilly Media, the Electronic Frontier Foundation (EFF), Nilay Patel of The Verge, David Drummond of Google, most of the US House of Representatives, etc.) so reform might be an easier goal.

I've co-opted some people's ideas on reform and come up with a few ideas of my own.  My list of reforms that would fix the software patent system are as follows:

1. Software patents must be specific.
   No more language about "the machine", "network web sites", "memory bank portions" and other archaic language unused since the days of vacuum tubes for the sole purpose of obfuscation.  These things are "the computer", "websites", and maybe "areas in memory" (?) respectively.  (Though, really, hardware usually has no place in a software patent.  Usually, hardware is only mentioned as part of a patent to lend credence to the notion that it is an "invention.")  Requiring clear and current language without extraneous implementation details will make patent application, processing, and litigation go a lot faster and easier.
2. Software patents must be used.
   If you are the owner of the software patent, you must be using the invention to actually create a product, or you must license the patent at a reasonable price.  Waiting for someone to violate your unused patent so you can extort them is unethical and it should be illegal.
3. Software patents should have a short lifespan.
   Twenty years is too long to hold a monopoly on a technology in the software industry.  Bezos and O'Reilly suggested 3-5 years, which is much more reasonable.  The EFF also said a max of 5 years.
4. The US Patent and Trademark Office must remove bad patents from their database.
   The Office must go through their existing patent database and get rid of duplicate patents and other invalid.  The USPTO is mostly to blame for this fiasco and they should fix this.  Otherwise, how can inventors tell what inventions to avoid using and applying for?

With reforms such as these, the purpose of software patents would be restored.  Software inventions would be protected from stealing, but it would be easy for inventors to tell what patents existed and whether an idea was new or not.  The title of most innovative tech company would no longer depend on who has the biggest legal team.  Patent trolls would have a hard time existing.  And when the occasional invalid patent was granted, the owner would only have an illegal advantage for a short period of time.

In summation, we Americans must reform software patents--or better yet, getting rid of them altogether.  We can fix the problems in the industry before any more damage is caused.  When we do, society will reap the benefits.

* The European Patent Office has allowed some patents involving software, but "programs for computers" have not been regarded as patentable inventions since 1973.

Policy: The Software Patent System Will Always Be Broken

(Our Broken Software Patent System, Part 2 of 3)

The software patent system is broken.  You know it, I know it, most software engineers know it, and even the government is vaguely aware that something is wrong.  But someone will fix it, right?

Not likely.  While I sincerely hope that with enough reform, the patent office can fix this mess of a system, I believe that the software patent system will always be broken.  Let me explain.

Reforming the System Is Hard
Changing such a huge system to be efficient and coherent would be hard.  The software patent system is at the intersection of law and software engineering, an area few people understand well.  As such, it can be difficult to solve its problems or even describe what the problems are.

There's a lot of stakeholders in the current system that need to be considered.  For example, can software patents be reformed while leaving hardware patents alone?  How do you convince hundreds or thousands of lawyers, judges and programmers to change what they've been doing for decades?  Is it possible to banish patent trolls without screwing over regular inventors, especially those that don't know how to sell their invention?

Changing the system will require a lot of precise, sweeping changes.  It's possible, but incredibly difficult.  So who's going to do it?

The Government Won't Fix the System
Firstly, the United States Patent and Trademark Office is unlikely to fix the software patent system.  For one, they are overburdened with work already.  In 2011, Chief Judge Paul R. Michel (retired) noted that, "Inventors must now wait three years, on average, and often longer. Thomas Edison had to wait less than three months. In my view, eighteen months should by the longest time anyone must wait."  He blamed the problem on lack of funding, which is needed for "thousands" of more patent examiners and better IT systems.

Whatever the root cause, the USPTO has too many patents to process and too little time to process them.  I suspect that this extra amount of work has lead patent examiners to approve lower-quality patents.  Patents that would not have been approved in less-busy times are now being approved.  A recent study also suggests that the patent office has lowered standards to cope with the huge backlog of pending patent applications.  The USPTO cannot fix the problem with duplicate and obvious patents being approved because they are too busy to examine applications carefully.

Perhaps the USPTO doesn't care about reforming the software patent system.  Surely, it would lessen their workload if the system was more efficient because there would be less patents to approve and less patent lawsuits to handle.  However, it could make their jobs harder by requiring patent officers to go through lots of existing invalid patents in the system.  But even if the USPTO doesn't look back at already approved patents, patent officers will have a harder job if patent applications require higher scrutiny instead of the rubber stamping the patent office does now.  "Rubber stamp" may seem like a bit of hyperbole, but the acceptance rate of patent applications is around 90%, when you don't count refiled applications as separate applications (see graph below).  If you try and try again, your application will be accepted, it seems.

The patent allowance rate has skyrocketed in recent years. Source: Ars Technica 

Like many inefficient government agencies, the problem may be with bureaucrats.  A political reason is needed to enact any major reform.  Until software patent reform becomes an election issue, we shouldn't expect much change from the USPTO.

Congress or the President could fix the system but, as I said above, patent reform is not a sexy issue and is unlikely to get the attention it deserves.  Furthermore, I doubt the President or Congress understand this thoroughly complex problem.  (Note that the House surprisingly passed the Innovation Act, which was a start, but the Senate shut it down earlier this year for some reason.)

The Industry Won't Fix the System
I doubt that software companies will fix the system.  Large software companies already have purchased hundreds or thousands of patents and are immune to patent trolls.  They can even troll smaller companies with their large portfolios of patents.  Small software companies are feel the most pain from this broken system, but they don't have the lobbyists to effect change in Washington.

Patent lawyers, whether they work for the government as examiners and judges or for companies as patent attorneys, have even less incentive to change the patent system.  They have become a hot commodity in recent times.  The broken patent system is their livelihood, and a lucrative one at that.

The System is Intrinsically Unfixable
Another reason I believe the software patent system will always be broken is because the idea of software patents is (pardon the pun) patently absurd.  Software is unique and is inherently unpatentable.  That's just one more reason software patents will never be "fixed."

I know this is a bold claim, but I'm serious.  Let's break it down.

Firstly, software is a fast-paced industry.  The next big thing is often deprecated in a short amount of time.  Let's look at smartphones for an example.  Although 3G wasn't new, 3G data was a huge selling point for the iPhone 3G in July 2008.  Two years later, HTC released the EVO 4G, the first 4G smartphone in the US.  However, that phone used WiMAX networking for its 4G capability, and WiMAX was declared dead as of 2013 or 2014.  All major US networks now use LTE for 4G data.  In less than 6 years, the way smartphones sent data over mobile networks changed multiple times.  Yet software patents last 20 years.

Think about how long 20 years is in terms of software.  It's 2014 (as of press time here at Omari's Tech Blog).  In 1994, Windows 3.1 was the most popular desktop operating system.  Nobody you knew had a cell phone, though I knew a few lucky families with a car phone.  You got on the Internet by dialing into one of AOL's servers.  My, how things have changed.

Twenty years is several lifetimes in software years.  This means that software companies must license any important patented technology if they are to survive, and that license may not priced reasonably or even available to license at all.  Twenty years is just way too long to give exclusive rights to a software invention when you consider how fast the industry moves.

A second issue is that patents aren't as necessary for software as they are for hardware.  I understand the need for protecting hardware.  Inventing new mouse traps or lawnmowers or killer robots requires not just time, but materials.  That first mouse trap may not work correctly, so you need to create lots of physical prototypes until you get it right.  That requires real materials, like wood and metal.  And probably more than a few mice.

"Inventing" software requires none of that.  All you need is one basic computer to program on and you can create endless prototypes without worrying about materials or trial runs in harsh conditions, unless you're trying to program for some specific device.  Even then, there are emulators that allow you to program apps for smartphones and vending machines, etc. without actually purchasing the device the software will end up on.  Technically, you don't even need the computer.

Lastly, the complexity of software causes difficulty in deciding their patent eligibility.  In Patent Failure: How Judges, Bureaucrats, and Lawyers Put Innovators at Risk by Bessen and Meurer, they describe how software's complexity hinders its patentability: "It is possible, however, that features of software technology make it particularly susceptible to the patenting of obvious ideas, especially given the legal doctrines of non-obviousness developed by the Federal Circuit."  Because software patents are quite susceptible to obvious patents, patent officers and patent judges have a much harder job.

Software is different.  Software is special.  Software might even be unpatentable.  But even if it isn't, the software patent system will always be broken because no one can fix it.

That's why we must abolish software patents.

Continued in We Must Abolish (or Reform) the Software Patent System.

Policy: The Software Patent System Is Broken

(Our Broken Software Patent System, Part 1 of 3. Or you can read the short version)

The American patent system is completely broken, at least when it comes to software patents. Billion-dollar software companies are suing each other over minor pieces of code, patent trolls run rampant, and consumers are losing out as the original purpose of software patents have been lost. In this three-part article, I'll first convince you that the software patent system is broken in the US, then that the system shows no signs of changing, and finally I'll discuss what we can do about it.

Note that this article is about software patents in particular. The US patent system may work better for other industries; I don't know. My research has been confined mostly to software patents.

The Point of Software Patents

Let's first talk about the point of software patents. Patents are a form of intellectual property, like copyright. Unlike copyright, patents cover inventions instead of creative works and last for a shorter time (20 years compared to copyright's roughly 100 years). The purpose of a patent is laid down in the constitutional clause that gives Congress the power "[t]o promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries" (Article I, Section 8, Clause 8).

The idea behind patents is great. If you spend years inventing something, perhaps building a better mousetrap, then someone shouldn't be able to see your mousetrap in a store, steal your idea and sell the same mousetrap while bypassing all the hard work you did refining the improved mousetrap.  Or worse, they could see your mousetrap in your secret lab while you are still refining it, steal your idea and beat you to market.  With patents, you must reveal your invention publicly, but then you have the exclusive rights to that invention for 20 years.

However, there are limits. You can't patent just anything, someone would patent the wheel and sue all the car manufacturers.  Patents must be a patentable thing (a process, machine, "[article] of manufacture," and composition of matter), and they must be  new, useful and nonobvious. "New" means you can't patent the wheel; it's already been invented. "Useful" means the invention has to actually do something. "Nonobvious" means you can't just slightly tweak someone else's invention; if there's already a patent on a standard pencil, the patent office won't grant you a patent on a slightly smaller pencil.  That's just an obvious extension of an existing invention.

Applying these principles to software is pretty new. Software patents only became legal in the US in the 70's and 80's. The courts are still trying to figure it out. In eBay Inc. v. MercExchange, L.L.C. (2006), Justice Kennedy and other justices questioned the wisdom of permitting injunctions in support of "the burgeoning number of patents over business methods," because of their "potential vagueness and suspect validity" in some cases.

Lots of Existing Patents Are Invalid

Non-New Patents
Now that we've discussed how software patents are supposed to work, what's the problem? Well, one problem is that people are patenting things they shouldn't be able to patent. Remember that all patents must be new, useful and nonobvious.  Well, people are patenting existing software inventions.

Two excellent sources of information about our broken software patent system are the This American Life episodes " When Patents Attack!" and " When Patents Attack ...Part Two". I cannot recommend these episodes of This American Life highly enough. You can listen to the first one below. But to get to my point, during Part One the journalists from This American Life (TAL) talked to David Martin, the founder of M-CAM. M-CAM is hired by governments, banks, and other businesses to assess patent quality. M-CAM uses special software to search through all existing software patents to find patents that are essentially the same. Ideally, each patent should be unique. However, sometimes the US Patent and Trademark Office (USPTO) grants patents for things that have already been invented and patented. During the TAL episode, Martin looked up how many matching patents there were for one specific software patent. There were 5,303 matching patents.

"We thought that would be an anomaly. And then we were told, oh no, it's not an anomaly. That happens. [....]And as I've testified in Congress, that happens about 30% of the time in US patents."
- David Martin, (emphasis added)

The specific software patent that Martin was looking up with TAL was Patent 5771354 "Internet online backup system provides remote storage for customers using IDs and passwords which were interactively established when signing up for backup services". According to the owners, Intellectual Ventures, it covers upgrading software on your home computer over the Internet. For example, "when you turn on your computer and a little box pops up and says, 'Click here to upgrade to the newest version of iTunes'". When TAL looked at the text of the patent it seemed like it did a lot more than that. Having looked at it myself, I'd agree. It appears to cover doing any sort of data backup over the Internet, using an ID and password.

Drawing from Patent 5771354 showing
how all computer networks work

Didn't online backup solutions exist before 1999, when this patent was filed? Apparently. The Wikipedia article for cloud storage talks about AT&T's PersonaLink Services in 1994. This was a service for storing your PDA data. That sounds like an existing invention to me. But let's ignore that example for now. Martin's software found Patent 6003044 and Patent 5933653, which also cover backing up data remotely.  This invention has been patented many times over.  When TAL's Alex Blumberg expressed incredulity that 30% of US patents were redundant, Martin pointed out there was a patent on toast.

Patent 6080436 "Bread refreshing method" was issued in 2000.  It patents toast.  Don't believe me?  Go read it for yourself.  Granted, it isn't a software patent, but it shows the flaws of our patent system.

It should be clear now that the USPTO allows inventors to patent inventions that have already been invented, breaking the requirement that all patents be new.  There are duplicate patents abound.  Furthermore, I'd argue that even if an inventor receives the patent on something, it doesn't mean he or she invented it.  That person may have just been the first to file a patent.  Filing a patent is an expensive, time-consuming process.  According to the USPTO, it takes about 2 years for a patent to be processed.  The cost of filing a software patent is around $10,000 (UpCounsel, Richards Patent Law, IPWatchdog) when including legal fees, which might be a drop in the bucket for a major corporation but is expensive for a small startup.  Because of these barriers to getting a software patent, the actual inventor may not be the first to file a patent on a particular invention.

Obvious Patents
So software patents may be patented and re-patented.  But even "original", previously unseen inventions may have problems.  For example, they might be obvious extensions of existing patents.  As I said above, you shouldn't be able to patent a slightly smaller pencil.  The pencil has already been invented, so taking an obvious idea such as making slight tweaks to the size or color is not patentable.  Sure, that idea might be "new", but inventions must also be nonobvious.  But obvious software inventions are being patented every day.

As a software engineer and a person who is skilled in the field, I would describe Amazon's 1-Click patent as fairly obvious.  In short, it describes the ability to save your credit card information in a website so that you can purchase items with only one mouseclick.  I am not the only who believes this patent to be obvious.  As one blog pointed out, it's a "fairly broad concept" that the European Patent Office denied a patent to because it was "obvious to a skilled person".  To give another example, this patent caused the Free Software Foundation to boycott Amazon for a short time on what it called "an important and obvious idea for E-commerce".

There are other examples of patents on obvious "inventions."  Richard Stallman, a software engineer who is famous for his work on GNU, Emacs, and other free software, decries the obviousness of Patent 5963916, applied for in October 1996 in his text, "The Anatomy of a Trivial Patent".  Patent 5963916 ("Network apparatus and method for preview of music products and compilation of market data") covers listening to a preview clip of music on the Internet.  After posting a snippet of the patent, Stallman writes, "That sure looks like a complex system, right? Surely it took a real clever guy to think of this? No, but it took cleverness to make it seem so complex. Let's analyze where the complexity comes from[...]"  Stallman then proceeds to analyze each line in the first portion of the patent to point out how each aspect of the idea was already existing or, at least, very obvious at the time:

Now look at a subsequent claim: 

3. The method of claim 1 wherein the central memory device comprises a plurality of compact disc-read only memory (CD-ROMs). 

What they are saying here is, "Even if you don't think that claim 1 is really an invention, using CD-ROMs to store the data makes it an invention for sure. An average system designer would never have thought of storing data on a CD."

In case it's not clear, Stallman is being extremely sarcastic. Even back in 1996, CDs were commonly used for storing data.  This is one of the patents that Stallman calls "laughably obvious".  The problem is that the overly complex language of patents obscures the obviousness of the ideas.

This is a huge problem.  Lots of software patents in the US are actually invalid because they are obvious or already patented.  If David Martin's numbers are to be believed, there are hundreds of thousands of invalid software patents in the system.  But invalidating a patent means spending lots of time (years) and money (millions of dollars) in court.  Because of this, the original purpose of software patents--promoting progress--has been lost.

Software Patents Aren't Serving Their Purpose

The first problem with software patents is the barrier to entry.  That is, the money and time spent in getting a patent may create a burden for startups and solo software engineers.  As mentioned above, it can take around 2 years and $10,000 to get a software patent.  This fact alone may dissuade programmers from trying to create innovative new software that the public would benefit from.  However, this is a small problem compared with other software patent issues.

One of the most insidious problems with software patents is the menace of patent trolls.

Patent Trolls

Alaska Robotics - "Patent Trolls"

According to the most popular definition, a patent troll (also called a patent assertion entity) is a company that obtains patents--usually through buying them--and, instead of making products based on those patents, waits for another company to violate their patents, and then forces that company to pay licensing fees to use that patent.

Let's take the example used in the introduction of the "When Patents Attack!" This American Life podcast.  Since 1999, Jeff Kelling has been working for FotoTime, a small company that hosts a photo sharing website.  This was before major photo sharing sites like Flickr, although FotoTime wasn't the first photo-sharing website.  In May 2008, they received a letter from FotoMedia (not to be confused with FotoTime) that said FotoTime was in violation of 3 patents.  FotoTime was told to contact FotoMedia to arrange payment, or FotoMedia would take them to court.  Jeff's team looked up the lawsuit and noticed that FotoMedia had sent the letter to 130 other companies including Yahoo (Flickr), Shutterfly, Photobucket, and other companies, big and small.

There were a lot of reasons this was weird.  One was that FotoTime hadn't realized that they were violating any patents.  Whatever patents were being violated, FotoTime had evidently come up with the same idea themselves, without reading FotoMedia's patent or copying some FotoMedia product.  Another weird thing was that FotoMedia wasn't a competitor to FotoTime.  FotoMedia didn't have a photo sharing website.  Then when Kelling called FotoMedia to ask them which patents FotoTime was violating, FotoMedia wouldn't tell them.  Kelling said, "They said they wouldn't answer that until we got into court."

Kelling learned that fighting the patent would cost an estimated $2 to $5 million, which was "more than [FotoTime] could handle".  FotoTime settled with FotoMedia.  As part of the settlement, FotoTime is forbidden from saying how much they had to pay FotoMedia.

This is what patent trolls do.  They purchase a patent that is somewhat vague or perhaps even invalid, and instead of making some technology that uses the patent, they threaten to sue companies they think are using the patent.  They might try to sue hundreds of companies, as in the case of FotoMedia.  Some companies will pay them a fee rather than spend millions of dollars fighting the patent in court.  The patent trolls will then use that money to purchase more patents and sue more people, ad infinitum.

This isn't a business.  This is, as venture capitalist Chris Sacca described such practices on This American Life, "a mafia-style shakedown."

And patent trolls are growing bolder.  According to a Presidential study, the number of lawsuits brought by patent trolls more than doubled from 2010 to 2012, and these lawsuits accounted for 62% of all patent lawsuits in America in 2012.  Victims of patent trolls paid $29 billion in 2011.  Now in 2014, these numbers are probably higher.

The Damage from Patent Trolls
This patent trolling kills innovation in a few ways.  One problem is that software companies must spend money on legal fees instead of growing their technology.  It's impossible to know whether your company will be violating a patent or whether a patent troll will target your company, so it's now important to have a good legal team to either negotiate a settlement or fight the patent lawsuit if and when a troll sues your company.

Some startups may not be able to pay the legal fees from a patent lawsuit and may go under.  Jeff Kelling of FotoTime said that "The settlement they [FotoMedia] wanted to get was just enough to put us in danger, but not to close us [...]"  It seems that lawsuit came close to shutting down FotoTime.

Patent trolling can even create a chilling effect, causing programmers good ideas to never pursue their startup for fear of being sued by a patent troll.

Software patents no longer promote progress, if they ever did.  Now, software patents stifle innovation.  It's impossible to know whether a company is actually violating a patent because the patent office is a confusing mess, containing vague and duplicate patents.  And so any software company with a good idea can be sued at any time, by a patent troll claiming that software company is violating its patents.  Sure, the current software patent system is great for patent trolls, who create nothing and provide no benefit to society.  And this is the exact opposite of the purpose of the software patent system.

The system needs fixing.

Continued in The Software Patent System Will Always Be Broken

---

This American Life: When Patents Attack!

Software Engineering: Good Reads

I'm not an avid book reader, so I feel that my book suggestions should be taken seriously, since there are so few books I can really recommend.  I think there are a few software-engineering-related books that every good programmer should have on their shelf.  I will list these books, roughly in order of importance.  I've also listed each book's latest edition (that I know of).

The C Programming Language, 2nd Edition by Brian W. Kernighan and Dennis M. Ritchie

The C bible. C is still an important language for many programmers to learn, and this is the book to use to learn it.  Never before or since has a programming language had a textbook so thorough.  I only wish that Java or C++ had a similar textbook that I could recommend as highly.  Maybe it's because C is a small language, but this book works well whether you're learning C for the first time, or you just need a reference.  The only downside is that the latest version of this book covers ANSI C, instead of either the newer revisions, C99 and C11.  For that reason, if you're using a C99 or C11 compiler, this book is somewhat deprecated. C99 made a lot of important changes to the language.  However, the book is still a good starting point if you don't know any C (or, again, if you're using an older compiler).

Introduction to Algorithms, 3rd Edition by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, Clifford Stein

This book contains all the information you need to know about algorithms and data structures, including their pseudocode implementations and computational complexity.  I suggest it to everyone who is applying to Google as a software engineer.  The only issue is that it is too complete, and contains a lot of extraneous information.  It's not a book you read cover to cover.  Also, I'm not a big fan of their "pseudocode."  A more Java-like or Python-like language would be easier to read.  As a software engineer for any company that relies on speed for computing large amounts of data, you will need to know the most efficient way to solve your problem.  This book will help you get there.

Effective Java by Joshua Block

That's great that you can write a one-line method to sort an array.  Now, graduate from a programmer to a software engineer by learning why that is awful.  Effective Java tells readers about readability and other important issues that come up with writing code in a work environment.  Basically, it's a collection of tips for writing readable, maintainable Java code.  Do you swallow InterruptedExceptions, that is, catch them and then simply throw them away?  If so, Joshua Block will tell you why you're breaking his heart and the heart of everyone around you.  (For those who just want the answer, read this. Or the simple answer is that you may prevent a program from being killed when it needs to die.)

Software Project Survival Guide by Steve McConnell

This book is written more for managers and team leads, but is useful to all engineers.  It describes in detail the basic ideas and reasoning behind a waterfall software engineering process.  A lot of you may be using agile or scrum, but you can still incorporate most of the ideas in this book in your work process.  Testing, maintainability, requirements, tracking... this book has all you need to turn your cowboy coders into seasoned engineers.  Process, process, process.

Team Geek by Brian W. Fitzpatrick and Ben Collins-Sussman

This book written by two Google engineers seems to be written more for team leads, but, again, is useful to any software engineer or manager of software engineers.  It's all about the non-technical side of software engineering; the human element.  For example, they cover dealing with "poisonous" coworkers, building a strong team, how to have efficient meetings, and how to communicate and collaborate well.  This is the only book I didn't have to read for school or work that I'm recommending.  I think it's important because even a group of smart, trained software engineers can have interpersonal issues or leadership issues, and this book tries to train people to alleviate those issues.

Software Engineering: Automated Software Testing 101

My official title at Google is Software Engineer in Test Tools and Infrastructure, which means I spend most of my time writing automated test infrastructure and consulting with teams to help them test their products.  For the last couple years that I've been working in this role I've learned a lot about testing.  I'd like to disseminate some of this knowledge.

There are many ways to test your software--way too many to cover in one short blog post--, but some of them are more important and widely applicable than others.  In this article I'll talk about the major types of automated software testing and why they are important.

Why and When to Test
Firstly, why do we need to test software?  Software testing can sometimes seem less important than other tasks.  For example, is there any reason to spend expensive engineer-hours writing tests, when you already have design reviews, code reviews, and manual code inspection to assure product quality?  Additionally, the fact that bugs can still exist in well-tested code is disheartening.

Despite the drawbacks, testing should be done early and often.  Testing your software increases product quality (and in doing so, lowers maintenance costs), aids development speed (testing takes time, but bugs are easier to fix when found earlier), proves requirements have been met, and can reveal design flaws.  It's usually worth the time to write at least a few small tests.

"The problem with quick and dirty, as some people have said, is that dirty remains long after quick has been forgotten."
— Steve McConnell, Software Project Survival Guide

(By the way, the Software Project Survival Guide is a pretty good book for those in the software field. I recommend it for engineers and especially for project managers).

This article focuses on automated tests because it's my area of expertise, manual tests are pretty easy to figure out, and because automated tests have several advantages over manual tests: automated tests are generally faster to run, can be run more frequently when part of a continuous build, are easier to use for unit and integration testing, and free up engineers to do other things.  If you're writing a small program for yourself, relying solely on manual system tests is fine.  For major projects, you'll need automated tests.

What to Test
So by now you've been persuaded by my convincing words that testing should be done early and often. But what are we testing?  If we test everything, at what point do we write tests for tests?

Tests should be created for most things, especially complex and critical functionality.  If you've created a function that does nothing but add a digit to the end of a string, it's not critical to write a test for that function. Most other stuff should be tested.

While tests themselves can break, tests don't usually need their own tests because of their simplicity and because a false failure will be visible to your team when the test runs.  A falsely-passing test can be a disaster, but is an infrequent occurrence for well-written tests that were working to begin with.  The exception is to write tests for test infrastructure.  Sufficiently complex tests are usually built on some sort of testing infrastructure (JUnit, WebDriver, a fake database, etc.) that can easily contain bugs, and accordingly, they should be tested.

Who Does the Testing?
Who writes tests: software engineers or a QA/test team?  The answer is that it depends.  For small teams creating small projects, it is sufficient to have engineers test their own code.  After all, it can be difficult to obtain a test guy or gal, and there isn't much code to test, so writing tests is quick and easy.  But even for large teams creating large projects, it can be beneficial to have the programmers write tests.  They can gain insight into their own code, have a better idea of what can break, and are more familiar what the issue is when a test breaks.  As someone who writes tests for other teams' projects, I've often encountered teams that have no idea why a test is broken or how to fix it, even though they know the project code better than I do.  That issue diminishes when the team has a hand in writing the tests.

Alternatively, a separate test team gains a different perspective of the code.  Unlike the programmers that wrote the code, a test person's ego is not affected when a test finds a bug.  In fact, finding bugs is ego-boosting for the test team.  The drawback is that a test team may not write tests that sufficiently cover the weak points of the dev team's code.

So the answer is: either way is pretty good.  Get a test team when writing tests becomes burdensome for the project programmers.  This can happen when the software is extremely hard to test for whatever reason, or when it's complex enough to require special test infrastructure.

Test-Driven Development
So tests should be written early.  But how early?  Some programmers believe in test-driven development, where you write the test before writing the code.  I think that's a bit drastic, but not always a bad idea.  APIs and features may change slightly when actually writing code, so your tests will likely have to change anyway.  You don't necessarily need to write a test before the code is written, but you should have a test plan.  Writing tests should be done simultaneously when writing code; that's early enough to catch initial bugs and late enough to prevent overhauling tests.

Now, on to types of tests...

Unit Testing
First and foremost, if you're going to have any automated tests (i.e. you're not writing a small program for yourself), you should have unit tests.  The reason is that unit tests are the easy and quick to write and run, unlike other tests.  Unit tests are small tests, usually white-box style, that test a class, a few classes, or another small portion of code.  Using mocks or stubs is perfectly fine for unit tests.

Because unit tests are so lightweight and quick to run, they should be run often, preferably as part of a continuous integration process.  This allows the unit tests to catch newly introduced bugs quickly.  Several solutions for continuous integration exist, like Jenkins. (I have not used Jenkins myself).

Integration Testing
The downside of unit testing is that important parts like databases or dependencies on other binaries* are mocked out or nonexistent.  You don't get a good picture of how the software performs under real conditions.  In order to verify the different subsystems of a program, you should write integration tests.  Integration tests are large tests that test multiple binaries, or one binary that uses external resources.  Integration tests are helpful in catching integration errors (often API or design bugs).  Even if two pieces of a project are "bug-free," they may not integrate well, resulting in miscommunication that can only be caught by integration tests.  Integration tests can also find speed or memory issues that can't be found by simple unit tests.

Since integration tests are large and slow, it can be difficult to find a test framework that will bring up all the resources (a.k.a the environment) needed for the test and then tear them down after the test is done.  One workaround is to leave up servers or other external resources indefinitely and have the tests clean up any permanent effects in the environment after the tests finish.  This is a dangerous game, as the long-running environment can get into a bad state, giving inaccurate test results.  Sometimes the only way to run integration tests is to manually bring up and down the environment.  In this case, it might be worth it to manually run the integration tests as well.

System Testing
To get a complete picture of how the software will actually perform, system tests are required.  System tests are large tests, black-box style, that bringing up an actual environment and simulating an end-user using your product.  Not only do they help reveal memory and speed issues that may only occur in a real-world environment, they also may reveal UI and usability issues.

System tests are often run manually since it's hard to find good frameworks that can both setup a large environment and give human-like input (often by manipulating a GUI).  However, many GUI-manipulating tools exist.  I've used AutoIt to manipulate Windows GUIs with excellent results and I've also used WebDriver to manipulate webpages with very good results.  Combined with integration-test-style frameworks, you can potentially automate your system tests.

In summation:

• Write tests early and often
• Run unit, integration and system tests, automating them if you can
• Run automated tests as part of a continuous integration process if you can

Following these suggestions will help you get excellent code coverage and excellent feature coverage, which will result in stable, easy-to-maintain, and on-schedule software.

Update: Added What and Who testing sections.

*Binaries are individually-compiled programs. They may not do much alone, but work in conjunction with other programs to create useful output.  If they do useful work by themselves, they are standalone binaries, or executables.

Policy: Open Letter to the FCC

The FCC has established an email address (openinternet@fcc.gov) for public comments on their Open Internet proposal.  I sent the letter below.  All comments become official FCC proceeding documents, so they are publicly available via the web.



FCC Commissioners,

My name is Omari Christian and I'm a software engineer.  I have read and written a lot about net neutrality and I am very concerned about Tom Wheeler's new proposed rules for broadband.  Particularly troublesome is giving Internet service providers (ISPs) leeway to charge content providers in a manner that is "commercially reasonable."  There is no amount of money that is commercially reasonable. Once an ISP is allowed to charge providers for faster or better access to users, ISPs will become the arbiters of which businesses succeed on the Internet and which businesses will fail.  The business that will succeed will, of course, be partners of said ISPs and also a few large corporations that can afford the new ISP tolls.  This cannot happen.  This will break the way the Internet has always worked.  It will hinder the free market spirit of the Internet.  It will diminish free speech by hindering one of the most popular methods of communication.

Net neutrality is good for free speech and good for the US economy.  It helps small businesses.  And allowing ISPs to charge content providers for access to Internet consumers is not net neutrality.

There are several things that the FCC can do to make sure the Internet stays open.  First and foremost, the FCC must classify broadband Internet as a telecommunications service (i.e. a common carrier).  This will give the FCC power to regulate ISPs like other common carriers.  Secondly, the FCC should make sure wording includes mobile carriers.  Mobile users are becoming a larger part of total Internet users every day and they may one day become the majority, if they haven't already.  Mobile Internet is part of the Internet.  And, thusly, mobile providers (AT&T, Sprint, T-Mobile, and Verizon included) are also Internet service providers.

Lastly, the most effective way to make sure broadband Internet is keep fast and open is to declare it a public utility, like water and electricity.  This may be difficult to square with mobile providers, but it can certainly be done with cable and DSL ISPs.  Just like electricity and water, broadband Internet is expensive to rollout to neighborhoods and costly to maintain.  Most areas only have one or two providers because of this natural monopoly.  Fast Internet service has become a necessity in this day and age, becoming one of the most efficient ways to do business, communicate with friends, and disseminate information.

Let's keep the Internet free and open for everyone.  Thank you for listening.

Sincerely,
Omari Christian

Tech Support: Step-By-Step Procedure for Removing Viruses and Other Malware

So, you've got a virus. Or maybe you don't. All you know is you have a Windows PC and the damned thing isn't working like it should. I’ve had this issue countless times, and approximately twice I ended up having actual malware. There’s a few ways to proceed in a situation like this.

You can call tech support for help, but that costs money and takes time. Or you can download and install Trend Micro HijackThis, run it, and post the log on some computer tech support forum. That seems to work for a lot of people, but requires finding the right forum and waiting for kind computer geeks to solve your problem.

Finally, you can try to get rid of the malware yourself using a good malware removal tool. In this article, I’ll describe this option, taking you through my usual process of diagnosing and removing a computer malware from a computer.

"Malware" is any malicious software, including worms, trojans, and viruses. "Anti-virus" is used to refer to any software that can get rid of malware. For a complete dictionary of malware types, check out Viruses, Spyware, and Malware: What's the Difference?

Diagnosing How Bad It Is
First, turn on the computer. If the computer won't turn on--like nothing shows up on the monitor--then you're kinda hosed. It probably wasn't malware. Malware that can damage hardware or wreck your firmware (i.e. your motherboard BIOS) is rare. It's likely that your hardware is broken for other reasons, like being super old or you dropping your computer down a flight of stairs. Seek help elsewhere.

If the computer turns on, but won’t boot into Windows. Several things could be wrong. It could be your hardware, or perhaps malware just corrupted your Windows installation. In any case, that’s more advanced than what we’ll try to solve here.

If the computer turns on and boots into Windows, but you have weird problems there--popups asking you to buy something to “fix” your computer, your browser has been changed to a spammy homepage, your anti-virus won’t run, etc.--then you might have typical malware.

Removing the Malware Manually
When you start up your computer, even before you open your web browser, do you get popups telling you how to fix your computer or speed up your computer? Did you install the program that’s telling you this?

If you have some unknown program that’s telling you that it can fix your computer, you have malware. This is the most common malware I’ve seen in the last few years. One example I recently uninstalled from a friend’s computer was a piece of malware called PC Fix Speed.

image from malwaretips.com

PC Fix Speed and similar fake anti-virus programs want you to be concerned about fake issues with your computer so that you’ll pay them for the full version of their software or for customer support. This type of malware is called "scareware" because the intent is to scare you into paying for their services. God knows what happens after you pay them (is the “fix” a program that uninstalls their own malware?).

Getting the Fix
Download Malwarebytes Anti-Malware. This program’s preventive anti-virus capabilities may be mediocre, but its ability to find and remove existing malware is unparalleled. In the several years I've been using it, I've found no better product for removing existing malware on a computer.

If downloading the software using Internet Explorer doesn’t work, try a different browser. Once I had to deal with malware that had hijacked Internet Explorer, making visiting any website impossible. Firefox and Chrome are a bit more secure.

If you still can’t download Malwarebytes Anti-Malware for whatever reason, you’re going to have to use a different computer to download it. Transfer the installation file (usually named mbam-setup.exe or something similar) to a USB flash drive or external hard drive and then use that to transfer it to your malware-ridden computer.

Closing the Gates
Once I no longer need the Internet, I unplug network cables from my computer and turn off the computer’s wifi, if it has wifi, in case the malicious program is spyware. Spyware can upload files from your computer so that hackers can obtain your passwords, credit card numbers and other personal data that can be used for nefarious deeds. Turning off your Internet prevents this.

Installation
Install Malwarebytes Anti-Malware. If you’re really unlucky, the malware has hijacked your registry and taken control of executable files. Installation will not work, and instead a different program will open, or the installation file won’t open at all. This happened to me once. I couldn’t open any programs at all except for the malware.

Getting Control of Your Executable Files (If Needed)

1. Boot into Safe Mode (Directions) and log in to your account or the Administrator account.
2. Click the Start button and type regedit in the Search box.
3. Right-click Regedit.exe in the returned list and click Run as administrator.
4. If a popup asks you if you want the program to be able to make changes to your computer, click Yes.
5. Browse to the following registry key: HKEY_CLASSES_ROOT\.exe
6. With .exe selected, right-click (Default) and click Modify…
7. Change the Value data to exefile.

8. Browse to and then click on the following registry key: HKEY_CLASSES_ROOT\exefile
9. With exefile selected, right-click (Default) and click Modify…
10. Change the Value data to "%1" %*
(That's quotation marks, percent sign, one, quotation marks, space, percent, asterisk.)
11. Browse to and then click on the following registry key: KEY_CLASSES_ROOT\exefile\shell\open
12. With open selected, right-click (Default) and click Modify…
13. Change the Value data to "%1" %*
14. Close the Registry Editor and restart your PC.

Directions edited from Microsoft Support.

Running Anti-Malware

Malwarebytes Anti-Malware

Once Malwarebytes Anti-Malware is installed, run it by clicking Scan Now in the bottom-right corner. Scanning takes a while. Select and remove all malware that Anti-Malware finds.

Install an Anti-Virus
Now your malware is (hopefully) gone. If you don't have a anti-virus program already, install one to prevent this from occurring again. I use Microsoft Security Essentials, but I've used Avast and AVG in the past. If you don't mind paying some money, McAfee and Norton make anti-virus programs that are probably better than the free options I listed.

Good luck!