double encoding of UTF8 strings

Turns out I was focusing on the wrong leads. These strings became values in a hash which was run through JSON->encode before saving to db. What prompted the re-encoding was not the values of the hash, but the keys. These were string literals in my program which were not marked as UTF8. Perl looked at my keys, looked at my values, saw a mixed bag, and decided to run the whole thing through the shredder again, just to be on the safe side.

The solution was simple:
use utf8

Because my keys were source-code string literals, I want my source code to be UTF8. use utf8 assures that.

thanks to: http://perlgerl.com/2011/04/08/utf8-follies/

how to debug anonymous functions and coderefs in Perl 5

http://perldoc.perl.org/B/Deparse.html

use strict;
use warnings;
use B::Deparse;

sub debug_coderef {
    my $possibly_code = shift; 
    my $deparse = B::Deparse->new();
    if (ref $possibly_code && ref $possibly_code eq 'CODE') {
        print STDERR "CODEREF: ", $deparse->coderef2text($possibly_code), "\n";
    }  else {
        print STDERR "NOT A CODEREF: ", Dumper($possibly_code), "\n";
    }
}

debug_coderef(sub {print "a";});

perl XS adventures, episode 1

Perl XS is easy if you read the docs in the proper order. What is the proper order ? I have no idea, I kept at it until it clicked. Well, not all of it clicked, just enough to get some results.

Summary so far: managed to persuade perl to run my code from the .xs file, to compile my .c files and to call code in the .c files from the .xs file.

To have it work I kept it simple. I tried the not-simple version and you don't want to know the story.

Hints for the inpatient

Quick hints

• man 2 and man 3 are your best friends, if you don't know this already; for example

  man 2 readdir

  or

  man 3 strcat

• read perlguts before perlapi; better, do not attempt to read perlapi end-to-end, just consult it when you need
• normal C code in custom .c and .h files, which all got to the root of the project (where Makefile.PL is); it is possible to put them some other place but I have not found out how to change Makefile.PL to deal with it.
• include .h files in the .xs file
• the code that uses Perl types and functions stays in the .xs file; Perl-independent C code can go before the

  MODULE = My::Module  PACKAGE = My::Module

  line
  
• extra libraries added in the Makefile as obiect files in here

  OBJECT            => '$(BASEEXT)$(OBJ_EXT) blah.o', 
  

  or even better

  OBJECT            => '$(BASEEXT)$(OBJ_EXT) blah$(OBJ_EXT)'

• to raise exceptions use

  croak(const char * blah)

  instead of exit(1) in the code in the .xs files

How to change the generated Makefile

based on http://www.perlmonks.org/?node_id=664066

Add a new compilation target

sub postamble {
    # this should be empty 
    my $inherited = shift->SUPER::postamble(@_);

    my $extra = q{
clean_temp_files :
 if [ -d /tmp/blah ]; then rm -rf /tmp/blah; fi
};
    return $inherited . "\n" . $extra;
}

Modify an existing compilation target

I think all the sections in the generated Makefile can be intercepted and modified, but so far I have tried only with clean:

sub clean {
  my $inherited = shift->SUPER::clean(@_);
  $inherited =~ s/clean :: clean_subdirs/clean :: clean_subdirs clean_temp_files/;
  return $inherited;
}

Step by step instructions for the very patient XS Mechanics or XS Fun.

searching all skype messages on Linux

if you ever need to search skype messages:

sqliteman

the query is "SELECT author, datetime(timestamp, 'unixepoch'), body_xml FROM messages ORDER BY timestamp desc;"

the sqlite file is ~/.Skype/your_user_name/main.db

Might need to stop skype before looking at the file.

inject subroutines at runtime

Creating subroutines on the fly using Symbolic references

how to install DLangIDE on Debian Wheezy

how to install DLangIDE on Debian Wheezy:

 - get the source from here:  https://github.com/buggins/dlangide or

git clone https://github.com/buggins/dlangide.git
cd dlangide
dub run

This will fail with

"
Derelict: cannot load freetype shared library: Failed to load symbol FT_Gzip_Uncompress from shared library libfreetype.so.6
"

or

"
2015-02-23 16:16:06.462 E No font files found!!!
2015-02-23 16:16:06.463 E Currently, only hardcoded font paths implemented.
2015-02-23 16:16:06.463 E Probably you can modify sdlapp.d to add some fonts for your system.
"

or

"
Failed to load symbol SDL_HasAVX from shared library libSDL2.so
"




or

"
SDL_CreateWindow failed - cannot create OpenGL window: No OpenGL support in video driver
"

No worries, will fix them next.

 - set the LD_LIBRARY_PATH to include /usr/local/lib if it does not include it already

export LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH

 - install freetype from sources (configure, make, sudo make install) from here http://sourceforge.net/projects/freetype/files/freetype2/2.5.5/

 - install libxext-dev (required by SDL)

apt-get install libxext-dev

 - install libsdl2-dev from wheezy-backports: don't need it since we'll install from sources, but lots of other dependencies will be installed

apt-get -t wheezy-backports install libsdl2-dev

 - install SDL2 from sources (configure, make, sudo make install) from here https://www.libsdl.org/download-2.0.php

 - install ttf-dejavu ttf-dejavu-core ttf-dejavu-extra

apt-get install ttf-dejavu ttf-dejavu-core ttf-dejavu-extra

 - update dependencies

dub upgrade --force-remove

 - edit a file in DLangUI:

find the source for DLangUI in ~/.dub/packages/ and edit the file src/dlangui/platforms/sdl/sdlapp.d :

find the line "ft.registerFonts("/usr/share/fonts/dejavu/");" and add after it this line:

ft.registerFonts("/usr/share/fonts/truetype/ttf-dejavu/");


return to the folder with the sources for DLangIDE and run

dub build --force

and then

dub run

or

./bin/dlangide

and you should get the main window to open

Did not get further today, will update this post if I get into more snags.





To get autocompletion install dcd from http://d-apt.sourceforge.net/ .

format json with Gedit and Perl

How to prettify/format/indent json with Gedit and Perl:

install JSON::XS
go to Plugins and enable "External tool"

go to Tools -> Manage External Tools -> add new (the button in the bottom on the left if no icons are shown)
add a name, for example "Format JSON with Perl"
replace the contents of the text area on the right with

#!/usr/bin/env perl 

use strict;
use JSON::XS;
local $/ = undef;
my $content = <STDIN>;

my $j = JSON::XS->new->utf8->pretty(1);
my $output = $j->encode($j->decode($content));

print $output;

open your file, then Tools -> External Tools -> "Format JSON with Perl"


you can also set a shortcut in the Manage External Tools window

good answer to an internet rant : "Perl already won"

Perl Already Won

readable text by an experienced Perl developer

my perlbrew build options

mostly copied from Debian, as shown by

perl -V


here they are:

perlbrew install -j 5 -Dusethreads -Duselargefiles -Dcccdlflags=-fPIC -Duse64bitint -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Doptimize=-O2 -Duseshrplib perl-5.20.1


"-j 5" thanks to http://www.dagolden.com/index.php/1384/parallel-make-for-perlbrew/

fglrx (AMD proprietary driver) on Debian Wheezy on Athlon 5350 with AM1H-ITX

The open source radeon driver works fine, but if for some reason you want to install the proprietary driver the Wheezy default will not recognize the integrated graphics AMD Radeon HD 8400. Fortunately the version in wheezy-backports does.

Add this line to /etc/apt/sources.list:

deb http://ftp.debian.org/debian wheezy-backports main contrib non-free


then

apt-get update

apt-get install -t wheezy-backports fglrx-driver fglrx-control

aticonfig --initial

reboot

Yes, the reboot is required, the open source radeon driver needs to be blacklisted and the package fglrx-driver apparently does that for you.

convert .msg files from Outlook to a format readable on linux with a text editor

How to convert .msg files from Outlook to a format readable on linux with a text editor:

from http://www.matijs.net/software/msgconv/

cpanm Email::Outlook::Message

or

cpan -i Email::Outlook::Message

or on Debian and derivatives such as Ubuntu:

apt-get install libemail-outlook-message-perl

then

msgconv my_unreadable_binary_file.msg
less my_unreadable_binary_file.eml