Category Archives: Programming

Experiments with Getting Web Content Using ColdFusion and PHP

Share this:  

So you want to get content from another web site out there to use on your site. You may be doing screen scraping of a page out there… But a better use is to get info from some sort of a web API.

Case in point: Calling a URI from the U.S. Weather Service to get current weather information.

http://w1.weather.gov/xml/current_obs/KMDW.xml

… This URI will return weather data from Chicago’s Midway Airport

Using ColdFusion

Here is a ColdFusion file for getting info on the current weather in Chicago:

It works fine. I do any Ajax call to “weather.cfm” from my home page, parse of the current temperature, and weather description and display it on my home page.  Nice!

 

If I call “weather.cfm” directly, I get weather data back for Chicago’s weather:

Below you can roughly see how it displays this page in Safari:

If you were to view the source, this is approximately what you would see, XML output:

Again, I have a JavaScript routine that calls weather.cfm via AJAX and pulls data from the <weather></weather> and the <temp_f></temp_f> tags.

This was running in ColdFusion Developer on my iMac and I can access on my personal wifi network.

 


Using PHP

I also wanted to do the same sort of thing using PHP. I decided I would use cUrl. Here is the code I used in a file called “weather.php”:

Notice that the URL is the same one as I’m using in the ColdFusion example.

 

So what happens if I directly call this “weather.php” page I created in my Safari web browser?

Well, we get a page like the one displayed above. Bummer! This is running on a PHP MAMP server on the same iMac as the ColdFusion server is. And trust me, the ColdFusion server is not calling this URL with any special permissions!

This led me to suspect that there was something different about the HTTP Header being sent by the ColdFusion server than was being sent by the PHP server using cURL.

But how to figure out what ColdFusion is doing differently than PHP?  Create a new PHP page to call instead of calling the xml file…

 


 

HTTP Header Test Page

I was going to create a PHP page that would look at it’s HTTP Header values and output them to the page for me to be able to see!

Here it is:

 

Now, if we changed weather.php to point to this URL, what do we get?

Note that the ‘1’ at the bottom is an artifact of cUrl (unless you set the CURLOPT_RETURNTRANSFER option to true.

What about doing the same thing with weather.cfm ?

There is definitely a difference between the two. Both have the same value for “Host”. Not much else is the same! It could be that the PHP request has HTTP Headers that the server does not like… But I’m going with the assumption that the PHP request is MISSING one or more HTTP headers that the web server (w1.weather.gov) is expecting. So lets modify our weather.php file:

 

Notice above how I added a new block of code (lines 8 through 12). This is adding three headers to our HTTP Header: ‘User-Agent’, ‘Connection’, and ‘Accept-Encoding’. I saved my changes and refreshed this page.

BINGO! IT WORKED!

But is the server looking for all three of these headers?

I remove ‘Accept-Encoding’.  I refresh the browser.  It still works.

I remove ‘Connection’.  I refresh the browser.  It still works.

And (of course) I remove ‘User-Agent’.  I refresh the browser.  And of course it fails.

So, ‘User-Agent’ is the key. Currently, in our example, it is set to the value of ‘ColdFusion’. Because that is the value I got when running the ColdFusion page. But actually (of course) our page is a PHP page when is requesting the info.

I change the value of ‘User-Agent’ from ‘ColdFusion’ to ‘PHP’. I refresh the browser and it works.

I wonder, is:   w1.weather.gov   looking for specific values for this header, or just that the ‘User-Agent’ header is present in the HTTP header?

So, I change the value of ‘User-Agent’ to: ‘SugarBoogers’.  I refresh the page and it works! This means that the server (at least in this case) is just checking to make sure that the ‘User-Agent’ HTTP Header is present and has a value… but doesn’t care WHAT the value is (I’m sure that ‘SugarBoogers’ is not a common user agent to check for!

Wrapping It Up

You might be able to “screen scrape” a web page without custom setting any HTTP headers. But I suspect that if your calling some sort of XML feed, JSON resource, or web service URI, there’s a good chance that you will need to set the ‘User-Agent’ HTTP header in order to get it to work.

Any comments? Thoughts? Let me know.

Happy Coding!

 

Resources

Entity Render Experiment in CodePen

Share this:  

Here is a little something that I’ve been messing around with in @CodePen. It renders the block diagram of a “data model.”  If you’ve ever used a modeling tool like Erwin this will look familiar…

Screen Shot of Model Render Demo

This is an experiment in rendering a lot of UI elements (entities) without having poor performance. In that, it will only render to the DOM entities that can currently be seen in the view port.

Right now, by default, it generates 200 entities. Try 2000 or 20000, or any other positive integer value, and see how it performs!

There are still a few bugs in it (its rendering too many entities at a time for example), but I think it proves its certainly possible.

You can try it yourself on CodePen here:     https://codepen.io/orvilleChomer/pen/NyXppd

Let me know what you think.

Enabling F12 Developer Tools in IE

Share this:  

For some strange reason on my work computer the F12 Developer Tools option was disabled on my Tools menu in Internet Explorer (IE 11). Other devs on my team had access to it, but I did not.

Well, I found out, at least in my case, that if I did the following steps, I could enable this option:

  • Click the “Start” button in the lower left-hand side of your screen.
  • Select the Run… option.
  • Type in the following and hit the Enter key:      gpedit.msc
  • You may be prompted to enter your Window user name and password, if so, enter it in.
  • The Local Group Policy Editor will come up.
  • On the left hand side of this window is a tree navigation control.
  • Make sure that the Computer Configuration node is expanded.
  • Under that, expand the Administrative Templates node.
  • Under that, expand the Windows Components node.
  • Under that, expand the Internet Explorer node.
  • Under that, select the Toolbars node.
  • Select the setting:  Turn off Developer Tools
  • Click the edit policy setting link to the left.
  • Pick the Disabled option and Apply the changes.

That is it! After I did this the F12 Developer Tools option in IE was accessible to me!

Doing a Page Redirect from a Java Struts2 Action Class

Share this:  

struts2Logo

I began working on a web site written in Java using Struts2. I wrote a general purpose class to be used by the application. One method in the class was supposed to check if the user was logged in. If not, redirect to the logon page (I did NOT want to add a tag entry to every <action> block in the struts.xml file)!

I searched the web and did not find anything that worked quite right. Finally, after some experimentation, I got something working! Here is some sample code for you:

Helper Class myExample.java:

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.servlet.*;
import javax.servlet.http.*;

public class myExample {
   public void doARedirectToGoogle() {
      HttpServletResponse response = ServletActionContext.getResponse();
      
      try {
         response.sendRedirect("http://google.com");
      } catch (IOException e) {
         e.printStackTrace();
      } // end of try / catch
   } // end of doARedirectToGoogle() method
} // end of myExample class

Note the try/catch block must be in place in order for this to compile and work.

 

Struts2 Action Class:  demoPage.java:

import com.chomer.demo;

public class demoPage extends ActionSupport {
   public String execute() {
      myExample demo = new myExample();
      demo.doARedirectToGoogle();
      
      return "success";
   } // end off execute() method
} // end of demoPage class

 

Action Block added to struts.xml:

<action name="demoPage" method="execute" class="com.chomer.actions.demoPage">
   <result name="success">/pages/demoPage.jsp</result>
   <result name="error">/pages/demoPageErr.js</result>
</action>

 

Success Page… demoPage.jsp:

<%@ page language="java" contentType="text/html; charset=ISO-8859-1" pageEncoding="ISO-8859-1" %>
<html>
   <body>
      <h1>This page will come up if the redirect does not work!</h1>
   </body>
</html>

 

Failure Page… demoPageErr.jsp:

<%@ page language="java" contentType="text/html; charset=ISO-8859-1" pageEncoding="ISO-8859-1" %>
<html>
   <body>
      <h1>This page will come up if the redirect does not work AND there was an error!</h1>
   </body>
</html>

In the above example I made up some arbitrary packages names. You will have your own structure in place. If this works you should never see the success or failure page.

In a real world scenario, the redirect would having only if a certain condition was being met (such as the user is not logged in). If the user were logged in, demoPage.jsp’s contents would appear.

Time Off and Some Progress on my Own Blog Software

Share this:  

Yay! I am off until the end of the year! When it comes to vacation where I work its “use it or lose it!”

Blog Software
Its on Apache, using PHP, mySQL, and a .htaccess  file. After futsing about I got the .htaccess file doing what I want for my new blog software (I am no expert on this file by any means though).

The development environment is my iMac using MAMP. I’ve got it reading a site table in my mySQL database. This software will support multiple sites on one server!

Next I’ve got to look up how to use sub-string functions in PHP. I need to take the rootPath field value from my sites table and compare it with the beginning of the URI that the user typed in in their browser. If it matches, I will set the site id, if not, I want to have some sort of fall-out page.

Wishing for Memory Functions in JavaScript

Share this:  

At work I am working on a web app that is very data intensive. A whole lot is done on the client side in JavaScript. There are certain instances in tests I have done where I have run out of memory.

I can adjust parameters down until the errors go away but:

  • Am I getting the absolute best usage of my memory or am I “leaving my change on the table?”
  • What if a user’s PC has less memory than me, are they going to get out of memory errors?
  • What if a user’s PC has a lot more memory than I do… then they really will not be able to leverage that extra memory to their advantage.

I wish there were two JavaScript functions available for programmers to use:

  • totalMemory()
  • availableMemory()

The functions could be called something else, that’s fine. But if there were these functions and if they returned the number of bytes relative to what was accessible to that current instance of the web browser, that would be so useful! As far as I know, JavaScript does not have any functions like this. I’ve looked 🙂

Am I missing something? Are there functions or object properties that I’ve missed? If you are a web developer working on some data intensive stuff on the client side don’t you wish you had these functions available?

One More Memory Related Thing
One more thing regarding JavaScript memory usage… There is a piece of info that I do know is out there somewhere… but I have not found yet… How much memory do JavaScript variables use?

I assume strings  may use 1 byte per character (or do they use 2 bytes to support unicode). I assume booleans take the least amount of memory (possibly 1 bit)? There are integers and floating point values… how many bytes do they use each? What is the overhead number of bytes for each variable and for each element in an array? Knowing this information could be helpful in deciding how to store information in memory the most efficient way.

UPDATE:
Microsoft says that their floating point variables in JavaScript use 8 bytes (search for the word “byte” on the page).
see: http://msdn.microsoft.com/en-us/library/ie/7wkd9z69(v=vs.94).aspx

UPDATE  2:
This looks like an interesting JS library: something where you pass in a variable or JSON string and it returns how many bytes it takes up.  I have to investigate this further when I have some time!
http://code.stephenmorley.org/javascript/finding-the-memory-usage-of-objects/

I Began working on new Blogging Software

Share this:  

For all the benefits of using WordPress for my blog, there are many more draw-backs. Sometimes I wonder if I would have been better hand-coding my blog! Writing a post like this that only contains your basic text, its fine. You start wanting to embed pictures or perhaps videos and it behaves very badly. I lost a whole long post recently 🙁 .

So, over vacation I started filling out a notebook with designs for my own blogging software. I’ve been doing a lot of content management software at work so doing content management type stuff is fresh in my mind. My goal is to get off of WordPress and get off soon. I want to be able to put together posts efficiently. I want to spend my time writing and refining my posts, not struggling with the technical problems of the platform I am using.

The database I am using is mySQL. I was going to write the blog software in Coldfusion (since that is what I am most comfortable with). But I have decided to use PHP. All nice and open source! 🙂 I’m developing it in a web instance running on my iMac using MAMP.

Goal: That I have a local instance working with basic functionality by this Saturday.

Doing Math in Objective C

Share this:  


One thing you will find you need to do in any programming language is do math. Here are some useful math functions for the Objective C language. I give a little bit of a comparison to Visual Basic for those who know that language. They are provided in no special order:

  • fabs() – Gets the absolute value of the value in parentheses. Works like ABS() function in Visual Basic. fabs(-123) = 123, fabs(34) = 34.
  • ceil() – Find the ceiling integer. Kind of like rounding up. Same as INT(x+1) in Visual Basic.  ceil(3.1) = 4.
  • floor() – Find the floor integer. Kind of like rounding down. Same as INT(x) in Visual Basic. floor(3.6) = 3.
  • exp() – Find the exponential value.
  • pow() – Raise a number to the power. pow(3,6) = 3^6 in Visual Basic.
  • rand() – Generate a pseudo random number. Function is RND(x) in Visual Basic. A good discussion on random numbers in Objective C can be found here.
  • srandom() – Generate a seed for the random number generator. This is done with the Randomize statement in Visual Basic.
  • sqrt() – Find out the square root of a number. sqrt(x) = x^(1/2) in Visual Basic.

Book Review: RibbonX – Customizing the Office 2007 Ribbon

Share this:  

My Book Rating: 5 of 5 stars:

Power Users and Developers: Making customized Apps Using Microsoft Office
Those of us who have the responsibility of producing customized Excel workbooks, and Word documents that essentially have become small powerful applications in their own right are always looking for ways to make the tools we build in these Microsoft Office products more powerful, easy to use, and more like actual stand-alone apps. One of the ways I looked at enhancing the “tools” I created was by adding a tool bar to an Excel workbook for example. I first did this myself in Excel 2003. It was tedious but doable. One draw-back in the way tool bars were done back then is that the changes applied to the whole app (Excel or Word), and one screw-up on your part could truly clobber Excel or Word’s tool bars and menus system wide.

A New Way of Doing Tool Bars in Office 2007
In Office 2007, Microsoft changed how their user interface worked. And they changed the scope of the changes. If you were going to screw something up for your tool bars now it would only be with one document file, not the entire system. Customizing this tool bar was more flexible. You could do much more with the new tool bar than the old version (the new tool bar Microsoft calls a ribbon). The Technology behind the ribbon they call RibbonX. Read more

Javascript: The Good Parts – A Collection of Lectures By Douglas Crockford on the Javascript Language

Share this:  

A few years ago, I am not sure just when or how, I stumbled upon these videos on Yahoo where programming legend Douglas Crockford gave very interesting and insightful lectures on the JavaScript language. It has been a bit tedious to find part X of a lecture and I would think about creating a page on the web that embeds all these videos in a way that makes watching them easier to do. To curate this content as Robert Scoble might say. Well after all this time I am doing just that in this post.  I think that if you are a programmer and you use JavaScript in any fashion that this information is really useful. Both to learn new things, as well as for review. Read more

1 2 3