Here are instructions for setting up a simple bottle.py Python web application running on Dreamhost (shared) with virtualenv.
(more…)

Here’s a simple shell script that turns an Oracle query into a CSV file suitable for mailing.
(more…)

With SQLPlus

Log into Oracle with SQLPlus.$ sqlplus user@dbname
SQL*Plus: Release 11.1.0.6.0 - Production on Mon Apr 12 11:46:48 2010

SQL> password
Changing password for USER
Old password: <old pass>
New password: <new pass>
Retype new password: <new pass>
Password changed
SQL> CTRL-D
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options


With SQL Developer
To change your password in SQL Developer, log into the database in SQL Developer and execute the following statement:

alter user username identified by "newpassword" replace "oldpassword"


The quotes make the password safe for punctuation characters.

Below, a series of shell scripts that make my life easier. Don’t expect any great revelations or virtuoso bash scripting.
(more…)

These instructions assume you’re using a Debian/Ubuntu-based system.

1. Create a web service/app (i.e., a Maven CXF JAX-RS archetype, or Flask/Bottle/Django, etc.)
2. Install curl
   sudo apt-get install curl
3. Install nginx
   sudo apt-get install nginx-full
   which should include nginx-common as a dependency.
4. Do not start nginx yet.
5. Add the nginx load balancer configuration and mime.types to your web service project (called MyProject in the following examples)
   1. Copy the mime.types definition file (as an alternative to copying the nginx configuration to /etc/nginx)
      cp /etc/nginx/mime.type ~/workspace/MyProject/web/conf
   2. Copy and modify the following load balancer/proxy configuration into MyProject/web/conf/balancer.nginx
      
worker_processes 1;
    
events {
    worker_connections 64;
    # multi_accept on;
}
    
http {
    upstream service-backend {
        server 127.0.0.1:9260;
        # server 127.0.0.1:8102;
        # server 127.0.0.1:8103;
        # server 127.0.0.1:8104;
    }
    
    keepalive_timeout 300 300;
    charset utf-8;
    default_type application/octet-stream;
    ignore_invalid_headers on;
    include mime.types;
    keepalive_requests 20;
    recursive_error_pages on;
    sendfile on;
    server_tokens off;
    source_charset utf-8;
    gzip on;
    gzip_static on;
    
    log_format main '$remote_addr $host $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $ssl_cipher $request_time';
    
    server {
        listen 127.0.0.1:8100;
        server_name mydesktop.domain.com mydesktop;
        add_header Cache-Control public;
        access_log /var/log/nginx/access.log main buffer=32k;
        error_log /var/log/nginx/error.log error buffer=8k;
        expires max;
        root /opt/MyProject/web/html;
    
        location /svc/ {
            proxy_pass http://127.0.0.1:9260;
        }
    
        location / {
        
        }
    
        location /favicon.ico {
            return 204;
        }
        }
}
6. Create a directory to hold your static web content (HTML pages, images, JavaScript)
   mkdir MyProject/web/html
7. Create a placeholder HTML file in the static web content/html directory
8. Set your webservice to run on the /svc URI endpoint (to match the nginx configuration)
   1. Update the MyProject integration tests
      
<parameter name="com.domain.webservices.AppRest.url"
type="text" desc="Example service URL">
%<value>http://localhost:9260/MyProject/svc</value>
</parameter>

   2. Update the MyProject/web/conf/web.xml to map the Jersey web application to “/svc/*” (or whatever is appropriate for your app)
      <servlet-mapping>
      <servlet-name>Jersey Web Application/svc/*
9. Build your project
   cd ~/workspace/MyProject
mvn clean install
10. Install and start your app (under /opt, for instance)
11. Start nginx as root
    nginx -c /opt/Myproject/web/conf/balancer.nginx
12. Verify your deployment
    1. Check your placeholder HTML page
       curl http://localhost:8100/index.html
    2. Check your service health
       curl http://localhost:8100/svc/health

1. Get a copy of jsmin.c from http://crockford.com/javascript/jsmin
2. Compile jsmin with
   gcc -o jsmin jsmin.c
3. Put jsmin in your path (i.e., under ~/bin)
4. Minify JavaScript

jsmin < input.js > output.min.js

As a best practice, when modifying third-party code rename your JavaScript file to something like filename.domain.min.js.

Caveat: this isn’t about failover, security, or cloud computing.

Imagine you have 65 million registered users and you need to provide custom content for each (beyond “Hello, Bob”), say real time subscription content from tens of thousands of sources containing millions of posts.

Don’t try to show everything at once

You probably won’t be able to show everything to the user at once. Depending on your load and back-end systems, you may be lucky to simply let the registered user know that they have something to see.

Most of your users won’t be logged in

Most page views (at least to a homepage) won’t be on logged-in users: many will be first time users, or users who haven’t visited very often. You’ll need to hit your back-end servers with a specific ratio of total visits, something you can measure early, and use as a baseline for how many servers you need to scale.
(more…)

SQL Developer is a good tool for creating queries, but you should be using SQLPlus when trying to time queries. If you don’t already have SQLPlus installed you can get it from Oracle:

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linx8664soft-100572.html

You want Oracle Database 11g Release 2 Client (11.2.0.1.0) for Linux x86-64. Unizip the client in your home directory.

Make sure you have the following environment variables defined:


# these point to wherever you have Oracle installed
ORACLE_HOME=~/oracle/product/11.1.0/client_1
LD_LIBRARY_PATH=~/oracle/product/11.1.0/client_1
PATH=$PATH:~/oracle/product/11.1.0/client_1


If you plan to run PL/SQL DDL/DML scripts, you’ll also need a SQLPATH environment variable which points to a directory where you keep the scripts.

SQLPATH=~/sqlscripts


If you don’t already have your database defined in your /etc/tnsnames.ora file (assuming you’re using that path and not something under /home/oracle), you’ll need to add the following TNS configuration:


dbalias =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dbhost)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = dbname)
    )
  )


To run SQLPlus against your database as a specific user, use the following command line:

sqlplus username@dbalias


Enter the password when prompted and you’ll now be at the SQL> prompt.

To get script timings, enter the PL/SQL command:

SET TIMING ON


1. Thou shalt encrypt all external communications with thy users
2. Thou shalt encrypt some internal communications on behalf of they users
3. Thou shalt keep thy passwords and thy email addresses in distinct and separate stores, as if they were credit card numbers
4. Thou shalt require encrypted communication with client keys to retrieve passwords and email addresses
5. Thou shalt never accept an unhashed or plaintext password and thy client will never send one
6. Thou shalt treat users as salted hashes and never have immediate identification of any user or user action in thy systems
7. Thy password and username systems shall be accessible only by API or service call and shall be implemented as separate, distinct, and secured networks, achieving defense in depth
8. Email campaigns shall be built on salted hashes and only the emailer shall have access to user names and email addresses
9. Customer service systems shall be able to construct salted hashes from user information but shall not keep copies of user names, email addresses, or passwords
10. Thou shalt disable all default user ids, passwords, keys, and conveniences for thy databases, management systems, and third party tools
11. Thou shalt never need to send an email to thy customers informing them that their private information has been accessed

A TechCrunch opinion piece posted yesterday, “Pair Programming Considered Extremely Beneficial,” was very complimentary about Pair Programming, a practice in which two developers work together to build software, one driving (typing) and the other navigating (describing what needs to be done). The author even included an amusing anecdote about Guy Steele pairing with Richard Stallman and how intense that experience was.

Since starting work at Overstock in 2010, I’ve had the opportunity to pair on a lot of user stories. Depending on the team lead pairing was either more or less the norm (less on my current team) but the company does have an inviolable rule when pairing must take place: when you’re working on financially impacting code. I’d extend that to say that you should pair on anything that impacts your core business and could cause the company to lose or have to restate revenue.
(more…)