Git in theory

Many times alternatives for version control systems are on topic, Git is a viable candidate. Arguments are not enough, though. One must answer the questions regarding then whats and hows?
Let us lay out a project utilizing Git for versioning source code, artefacts and assets etc.

First thing to mention: its distributed nature is not the first thing to talk about. It is the core feature, but not the most important quality to consider first. At the beginning it should be shown beforehand that it is able to fulfill the requirements of version control at all.

Revisions

In addition to remote repositories, Git also treats each working copy as a repository, in fact, they are local repositories per se. This makes every desired state perfectly revisionable, even local ones. On the file system level, only one particular snapshot (e.g. development state) is available, but you can take arbitrary snapshots. Commit and update refers to the interaction within this local repository.

Branching

Having a desired changeset, diff to share, you can use the push and pull operations to branch and merge between repositories available on the network. So you can for example push your local changes to the repository you were forking it in the first place from, or you can pull and merge the changes available in between repositories.
Please note that we are still considering a centralized management of the branching strategy. I.e. there can be several team level but one project level branch, and there can be additional release branches for each released revision (and their follow up bugfixes), and there can be sandbox forks to play around, and so on, but all in all, there are rules for where you can push to and pull from, enforced by rules of thumb (a.k.a process), hierarchy of trust (a.k.a review) and access control (a.k.a roles).

Who does the pushing and pulling, and more importantly who resolves conflicts? Git allows a great deal of flexibility here:
You can have dedicated people responsible for pushing all the changes upstream or pulling all the changes from downstream, resolving conflicts as they arise. Or you can have each person responsible for pushing each change from downstream to the top by following the process, getting reviews and acquiring required roles, but also resolving conflicts as they arise.
And a mixed approach is also feasible where for example team level changes are usually not branched but just pushed into the master repository, albeit changes which should go into release branches are pulled by individuals responsible for the release branch.

Distributed

So wherein lies the distributed nature of Git? By treating everything as a repository and giving flexibility on how to distribute responsibilities, upstream and downstream become relative terms. For the individual a clear hierarchy remains, can pull safe changes, and push resolved revisions. On the team level sub-teams and ad-hoc working groups find it intuitive to share code at any time, or just to fork a sandbox repository which can evolve on its own later on. Git can basically do whatever you expect from a centralized revision control system, but it heavily supports branching and merging across revisions. It does not force you to look at the project from the same perspective.

The bad

If you need tools to work with Git, its command line client is indeed sophisticated and once getting used to the terminology it serves as the best Swiss knife. However, GUI tools and enterprise integration is evolving at its best, you better check your environment and requirements, especially paying attention to the needs of every single one on the project beforehand.

Views: 2300
Bookmark and Share
Posted in Hacks | 4 Responses

QAssert: JavaScript assertions with AJAX reporting!

QAssert is a powerful, easy-to-use, JavaScript assertion suite. In contrast to console.assert() it enables you to report failures via AJAX. It gives you also powerful assertion functions and is cross-browser compatible.
It can be used in both in unit, functional and manual tests, and notably also
in production code.

QAssert is especially useful for regression testing: You can place assertions for any assumption that is made in the JavaScript logic. While running the code assertion violations will be reported via AJAX and regressions become quickly visible. To give you an example how to safeguard against unexpected DOM changes:

var $panel = $("#menuPanel");
$panel.assert().load("/");
$.assert($panel.text().length);

It is a simple drop-in jQuery plugin. It is disabled by default. To enable it and to see assertions:

$.assertSetup("/assertReporter"); // Required only once.
$.assert(false)

As a result:

  • You will se the assertion failure in the console including a full stack trace (Assertion failed: false undefined {anonymous}()@file:///D:/Dropbox/dev/qassert/qassert/qassert.js:708 …)
  • You will see an AJAX call being made to the given URL

It is highly configurable:

$.assertSetup({
enabled: true,
ajax: "/fooBar", // You can also pass a full $.ajax() parameter object here!
catchGlobalErrors: true, // This will report ALL uncatched exceptions too!
contextCallback: function(value, message, stacktrace) {return {version: app.version, locale: app.locale}}, // Enhances the reported data
log: myLogFunction, // Used to log failures locally, by default to the console.
title: "Assertion sucked up:"
});

It has a convenient API:

  • $.fn.assert(message, sizeOrCallback): Chained selector assertions.
  • $.assert(value, message): Boolean assertion.
  • $.assertDeepEquals(value, expected, message): Recursive equality assertion.
  • $.assertEmpty(value, message): Empty assertion.
  • $.assertEquals(value, expected, message): Equality assertion, no recursion.
  • $.assertIs(value, type, message): Type assertion.
  • $.assertNot(value, message): Boolean assertion for false.
  • $.assertNotDeepEquals(value, expected, message): Recursive unequality assertion.
  • $.assertNotEmpty(value, message): Non-empty assertion.
  • $.assertNotEquals(value, expected, message): Unequality assertion, no recursion.
  • $.assertNotIs(value, type, message): Type assertion inverted.
  • $.assertNotSame(value, expected, message): Strict unequality assertion.
  • $.assertSame(value, expected, message): Strict equality assertion.

See the documentation and test cases.
Join the development: https://github.com/gaboom/qassert

Views: 1677
Bookmark and Share
Posted in Hacks | Leave a comment

Instant PHP eval

Here is the code you get a form with, you can submit php code, it gets evaluated on the server.
SECURITY WARNING: command(s) below will be all executed in PHP’s server context! Deny public acces.

<!doctype html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>PHP evaluator</title>
    </head>
    <body>
        <div>
            <?php
            $function = "eval";
            $arguments = "return true";
            if ($_SERVER['REQUEST_METHOD'] == "POST") {
                $result = null;
                $function = $_POST["function"];
                $arguments = $_POST["arguments"];
                switch ($function) {
                    case "eval":
                        $result = eval($arguments . ";");
                        break;
                    default:
                        $result = $function($arguments);
                }
                echo "<hr/>";
                var_dump($result);
                echo "<hr/>";
                print_r($result);
                echo "<hr/>";
                echo($result);
            }
            ?>
        </div>
        <hr>
        SECURITY WARNING: command(s) below will be all executed in PHP's server context! Deny public acces.
        <form method="POST">
            <textarea name="arguments" style="width: 600px; height:200px"><?php echo $arguments ?></textarea>
            <br/>
            Function: <input type="text" name="function" value="<?php echo $function ?>" />
            <button>EXECUTE</button>
        </form>
    </body>
</html>
Views: 1861
Bookmark and Share
Posted in Hacks | Leave a comment

How to write down passwords onto paper safely?

Writing down usernames and passwords with a pen is insecure in so many ways I do not attempt to count.
The sheer count of passwords however, forces most persons to store them somewhere, instead of keeping all of them just in your head.
The digital solution is to protect the passwords with another password in sort of a vault (e.g. in a file on a password protected disk, encrypted volume or file, password storage program) [Please note: browsers store passwords almost cleartext by default!]

Sometimes you are still forced to keep a password written up quick and easily in an unencrypted environment, like a piece of paper. (Developing and memorizing secret code language is rather difficult to accomplish this.)
The trick is to write down the password, but with a simple to remember twist!

For each password pair you can see first an example of a cleartext password, then the transformed version:

  • alaudZAPP1 – alXZaudZAPP12
  • persepoleS – pe9lrsepoleSq
  • Bo23LOwf3 – BobT23LOwf34

You can insert at the very same position(s) random letters to make the text written down unusable. In this example I choose to insert two the after the first two letters, then one at the end.
Looking at the passwords on the right, remembering your scheme, you can easily re-type (decode) the original passwords but leaving an attacker with an obfuscated one.

WARNING: This method is by far not as safe as a fully encrypted password, and offers only a better-than-nothing protection. Common patterns in your password will become visible by this! Never re-use a password, this method reveals any such mistake, also making it obvious how you encoded all your other passwords written down.

Views: 1439
Bookmark and Share
Posted in Hacks | Leave a comment

Three questions for an architect

For a release sign-off

  • Do all unit, automated and regression tests run?
  • What is the status of outstanding bugs?
  • Are all changes demonstrated to stakeholders?

For a design sign-off

  • Are all transformations from the input domain to the output domain possible? Identify first risky and/or complex ones.
  • Are all our needs supported by an API?
  • Is our API supporting all requirements?
  • BONUS: If it takes more than two days to complete, does a task breakdown exist?
Views: 1157
Bookmark and Share
Posted in Hacks | Leave a comment

Business Software Development

Business Software Development
Eötvös Loránd University Budapest, Faculty of Informatics
Eötvös Collegium Informatics Workshop 2011-

Syllabus

  • What is business, software and development?
  • Goals, realization and measuring success. Sustaining a business.
  • Software lifecycle, development as a process.
  • Planning and executing projects: Lean, Agile, Scrum.
  • Architecture, design and documentation.
  • Requirements: functional and non functional. User interaction and interface design.
  • Version control and configuration management. Integration and release.
  • Testing and bug tracking, support and customer awareness.
  • Understanding different company cultures and roles.
  • Copyright and intellectual property.

Literature

Real Software Engineering [Glenn Vanderburg; Lone Star Ruby Conference; 2010]

Code as Design [Jack W. Reeves; Developer.*; 1995]

No Silver Bullet: Essence and Accidents of Software Engineering [Frederick P. Brooks, Jr.; Computer Magazine; April 1987, University of North Carolina at Chapel Hill]

Design Patterns: Elements of Reusable Object-Oriented Software [Erich Gamma et al.; Addison-Wesley Professional; 1994; ISBN: 978-0201633610]

The Mythical Man-Month: Essays on Software Engineering [Frederick P. Brooks; Addison-Wesley Professional; 1995; Anniversary edition; ISBN: 978-0201835953]

The Pragmatic Programmer [Andrew Hunt, David Thomas; Addison-Wesley Professional; 1999; ISBN: 978-0201616224]

Views: 1601
Bookmark and Share
Posted in Teaching | Tagged | Leave a comment

How to archive Drupal as a static site?

Here is the summary how I’ve managed to archive my Drupal weblog before migrating to WordPress. To do this, you should never put your blog/site to the root of your URL, i.e. example.com/ .
Instead, put it under a directory like example.com/foo so you can it archive it under this address and put up your new site at example.com/bar . The root address should always redirect to the current site.
Thanks to http://drupal.org/node/27882 and http://www.starbowconsulting.com/blog/tao/replacing-live-drupal-site-static-archive .

Steps to make the site ready being archived:

  • Disabled all commenting with update node set comment = '1'
  • For each content-type, set the comment controls to “Do not display.”
  • Disable search completely, also in the theme
  • Remove all dynamic blocks, login, register, who’s online, actual links, feed links etc.
  • Put up a static block and a sticky post stating this is a static archive with a link to the new one.
  • Remove all spam comments and spam links before archiving or search engines will ignore you.

Then I used httrack to archive my site:
httrack http://example.com/foo -W -O "~/static_archive" -%v --robots=0

That’s it. Double check your logs, I’ve had some old nodes giving a server error because of an outdated plugin.
Now you can remove your Drupal site and drop the corresponding tables. But there is one thing left:
If you used pretty URLs, like /node/4, you will need to set up redirects to show the archived html files instead (/node/4.html).
You might also want to redirect the RSS feeds to migrate your RSS subscribers to your new site and its feed.



RewriteEngine on
# If we are accessing root, redirect to index.html
RewriteCond %{REQUEST_URI} ^/foo/$
RewriteRule ^(.*)$ index.html [L]
# Redirect RSS feeds
RewriteCond %{REQUEST_URI} ^/foo/rss.xml$
RewriteRule ^(.*)$ http://example.com/foo/feed/ [L]
RewriteCond %{REQUEST_URI} /foo/feed$
RewriteRule ^(.*)$ http://example.com/foo/feed/ [L]
RewriteCond %{REQUEST_URI} /foo/.*/feed$
RewriteRule ^(.*)$ http://example.com/foo/feed/ [L]
# If we are accessing a page without a "." in it, append .html to the end.
RewriteCond %{REQUEST_URI} !\.
RewriteRule ^(.*)$ $1.html

Views: 13965
Bookmark and Share
Posted in Hacks | Tagged , | Leave a comment

Import XLS/CSV into MS SQL

What about the simple task of importing a tabular Microsoft Excel Sheet into a Microsoft Transact SQL table?
After paying thousands of dollars in license fees, this will take you a day! And you will need OpenOffice to do that :)

First google “import xls into mssql” will give you a simple solution:

SELECT * INTO XLImport5 FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0',
'Excel 8.0;Database=C:\test\xltest.xls', 'SELECT * FROM [Customers$]')

That will not work! Jet provider is not supported on 64bit! Yeah, Microsoft is not supporting 64bit users of its enterprise database solution. Thank you very much. (Just google how many ppl are ranting about the lack of this…)

OK! No problem I’m going to write a program that generates INSERT INTO statements from a CSV export, here you go:

package eu.gablog.csv2sql;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.text.DecimalFormat;
import java.text.DecimalFormatSymbols;
import java.text.ParseException;
import java.util.LinkedList;
import java.util.List;

import au.com.bytecode.opencsv.CSVReader;

public class CSV2SQL {

    private static final boolean truncate = false;
    private static final String src = "D:\\tmp\\src.csv";
    private static final String dst = "D:\\tmp\\dst.sql";
    //private static final String DB = "HCOM_EMEA.dbo.HCOM_BOOKING_POSTPAY_MAPPING";
    private static final String DB = "MHOTEL.dbo.PROPERTY";

    // GENERIC VERSION: first row contains column names
    public static void main(String[] args) throws IOException {
        // PARSE ALL ROWS
        CSVReader csvRows = new CSVReader(new FileReader(src), ';', '"', 0);
        String[] nextLine;
        List rows = new LinkedList();
        while ((nextLine = csvRows.readNext()) != null) {
            rows.add(nextLine);
        }

        // PREPARE OUTPUT SQL
        BufferedWriter out = new BufferedWriter(new FileWriter(new File(dst), false));
        if (truncate) {
            out.write("DELETE FROM " + DB + ";");
            out.newLine();
        }

        // INSERT STATMENT
        int rowCount = 0;
        String insertInto = "";
        for (String[] row : rows) {
            if (rowCount == 0) {
                insertInto = parseColumns(row);
            } else {
                String insert = parseRow(row);
                out.write(insertInto + " VALUES " + insert + ";");
                out.newLine();
            }
            rowCount++;
        }
        out.close();
        System.out.println("DONE: " + rowCount + " lines parsed from " + src + " to " + dst);
    }

    private static String parseColumns(String[] columns) throws IOException {
        StringBuilder result = new StringBuilder();
        result.append("INSERT INTO " + DB + " (");
        boolean first = true;
        for (String column : columns) {
            if (!first) {
                result.append(", ");
            }
            result.append("[" + column + "]");
            first = false;
        }
        result.append(")");
        return result.toString();
    }

    private static String parseRow(String[] row) throws IOException {
        StringBuilder result = new StringBuilder();
        result.append("(");
        boolean first = true;
        Double d;
        for (String column : row) {
            if (!first) {
                result.append(", ");
            }
            if (column.isEmpty()) {
                column = "NULL";
            }
            if ("NULL".equalsIgnoreCase(column)) {
                // NULL
                result.append(column);
            } else if (isLong(column) != null) {
                // LONG
                result.append(column);
            } else if ((d = isDouble(column)) != null) {
                // DOUBLE
                result.append(d.toString());
            } else {
                // STRING VALUE
                result.append("'" + escape(column) + "'");
            }
            first = false;
        }
        result.append(")");
        return result.toString();
    }

    private static Long isLong(String str) {
        try {
            return Long.parseLong(str);
        } catch (NumberFormatException nfe) {
            return null;
        }
    }

    private static Double isDouble(String str) {
        try {
            DecimalFormat df = new DecimalFormat();
            DecimalFormatSymbols dfs = new DecimalFormatSymbols();
            dfs.setDecimalSeparator(',');
            df.setDecimalFormatSymbols(dfs);
            Double d = df.parse(str).doubleValue();
            if (!d.toString().replace('.', ',').equals(str)) {
                d = null;
            }
            return d;
        } catch (ParseException e) {
            return null;
        }
    }

    private static String escape(String str) {
        String result;
        result = str.replaceAll("'", "''");
        return result;
    }

}

You think you can use that just this easily? Of course not! Microsoft Excel does not know anything about UTF-8 in 2010 AD!
CSV export has some random encoding just to fuck with you for the fun. No option for having it in UTF-8.
Just fire up OpenOffice, go to Sava As, and enjoy the beautiful options of selecting the export’s encoding. Now use the converter on this input.

Views: 3753
Bookmark and Share
Posted in Hacks | Tagged , , | Leave a comment

Captcha for CakePHP 1.3 with KCAPTCHA

We are going to use KCAPTCHA to generate Captchas and verify them in form submissions in CakePHP. Look at phpinfo(), you will need

First, download KCAPTCHA and put the directory “kcaptcha” from the .zip into app/vendors/.

Insert this into any view you want to have a captcha on:

captcha
input('captcha', array('label' => __('Nem vagyok robot:
', true))); ?>

We load the image with a semi-random URL, ensuring to always load a new one. In my case, it will be loaded from AjaxController:

function captcha($random = null) {
        $this->layout = null; 
        $thiy->$autoRender = false;
        //$this->RequestHandler->setContent('jpeg', 'image/jpeg'); // use this if your mime-type is incorrect

        App::import('Vendor', 'kcaptcha/kcaptcha');
        $kcaptcha = new KCAPTCHA(); // renders the captcha image fully here
        $this->Session->write('captcha', $kcaptcha->getKeyString());
    }

As you can see, displaying a captcha by fetching it, immediately stores the captcha’s solution in the ‘captcha’ session variable.

When you process the submitted form, where you have included the captcha, use this (Component) function to check the submitted value against the one stored in the session.

function checkCaptcha($text) {
        $captcha = $this->Session->read("captcha");
        $this->Session->delete("captcha"); // SECURITY: must clear captcha to avoid repatcha
        if (empty($captcha) || $captcha != $text) {
            return false;
        }
        return true;
    }

Be aware, that upon checking, regardless of the match, the captcha must be destroyed (removed from the session). Otherwise recaptcha is possible (re-submission over and over again with a first valid captcha).

At last, kcaptcha_config.php to finetune your images.

Views: 3835
Bookmark and Share
Posted in Hacks | Tagged , , | Leave a comment

Twitter OAUTH

Twitter will phase out the widely-used HTTP basic authentication by August 2010. The amount of working examples online makes me believe that most applications that make use of the Twitter API is not migrated yet. Even the simplest action will requires to use OAUTH.

This tutorial will guide you through the authentication process. In the end of our example we will get hold four secret keys in our hands that can be used at any later time to access the API with the same credentials again. You can download all files here. (You’ll need CURL to be enabled in PHP.)
The walk through corresponds to the “Desctop Clients” scenario in the first link.

1. Go to http://twitter.com/apps/ and click “Register new application”. For Application Type choose Browser and for the callback URL, the URL of confirm.php.
Now you receive a Consumer Key and a Consumer Token. Put them into secret.php.

2. Fire up index.php. You can double check your Consumer credentials here. First you’ll see only one link to authorize via twitter. Follow that, confirm the access.

3. Twitter’ll redirect you back to confirm.php. But you also gained now the Provider Token and Secret! Put them as printed on the screen into secret.php.

4. Going back to index.php you’ll see that you have no all the keys you need. You can check it on random.php, that uses them. Store this four keys securely. They won’t be revoked, unless the twitter profile explicitly removes the app. It’ll work even after changing the password.

Having just this four keys, using the same API classes, updating the status is as easy as this:

twitterObj = new EpiTwitter($this->consumer_key, $this->consumer_secret, $this->provider_token, $this->provider_secret);
    }

    /** Tweets the message with '...' shortening. If $url given, appends it.
     *
     * @param  $message
     * @param  $url
     */
    function tweet($message, $url = null) {
        $length = $this->maxLength - (empty($url) ? 0 : mb_strlen($url)+1);
        if (mb_strlen($message)>$length) {
            $length -= 3;
            $fix = "...";
        } else $fix = "";
        $message = mb_strcut($message, 0, $length).$fix;
        if (!empty($url)) {
            $message .= " ".$url;
        }
        return $this->twitterObj->post_statusesUpdate(array('status' => $message));
    }

}
?>

If you don’t store the Provider’s token and secret, instead storing it in the session, you can use Twitter accounts to authenticate profiles on your site.

Views: 40776
Bookmark and Share
Posted in Hacks | Tagged , , | 6 Responses